Data Perspective

How to import data into Google Colab Jupyter Noteook

2019-02-23T11:25:00.001+04:00

Accesing data is one of the first step that we need when performing any data analysis. In this tutorial, we will see two ways of loading data into the google colab environment.

Uploading csv from local machine and loading into colab

Loading data from google drive to colab

Uploading CSV from local machine using IMPORT functionality.

Load import files library from google colab

upload file using the upload button control

Running below commands will allow us to upload data files into the colab environment. Once the Choose Files button is visible, after executing the below listed python commands, we can easily upload files from local directory.

from google.colab import files
uploaded = files.upload()

Saving DOLPHIN.csv to DOLPHIN.csv

To view the uploaded files

Below command allows us to verify if the file is uploaded correctly.

for fn in uploaded.keys():
  print('User uploaded file "{name}" with length {length} bytes'.format(name=fn, length=len(uploaded[fn])))

User uploaded file "DOLPHIN.csv" with length 117269 bytes

Reading the uploaded from into pandas dataframe and displaying results

After the data file is uploaded to the colab, we can use pandas functions to load data into python environment and continue our further analysis.

import pandas as pd
import io
df = pd.read_csv(io.StringIO(uploaded['DOLPHIN.csv'].decode('utf-8')))

print(df.head(2))

Load data from google drive:

Sometimes we may require to load data from google drive. Below commands will be useful in reading data from google drive. Here we assume that the data file to be loaded into python environemnt is already uploaded to Google Drive.

Step 1, we need to mount the google drive

Step 2, After mounting we need to provide authorization

Step 3, we can view the current list of files available at the mounted location

Step 4, Load data using pandas read_csv function

Go to this URL in a browser: https://accounts.google.com/o/oauth2/auth?client_id=947318989803-6bn6qk8qdgf4n4g3pfee6491hc0brc4i.apps.googleusercontent.com&redirect_uri=urn%3Aietf%3Awg%3Aoauth%3A2.0%3Aoob&scope=email%20https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fdocs.test%20https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fdrive%20https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fdrive.photos.readonly%20https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fpeopleapi.readonly&response_type=code
Enter your authorization code:
··········
Mounted at /content/gdrive

Note: My data files are located at MY DRIVE\COLAB NOTEBOOKS folder of my google drive account. Please change the code accordingly
After clicking the link and entering the authorization code, you can access your drive as follows:

!ls -la /content/gdrive/My\ Drive/Colab\ Notebooks/

total 230
-rw------- 1 root root 117269 Feb 23 06:43 DOLPHINOFFALLN.csv
-rw------- 1 root root 12104 Feb 23 06:46 ImportDatatoColab.ipynb
-rw------- 1 root root 8935 Sep 8 17:23 'Running first neural network model on google colaboratory'
-rw------- 1 root root 13498 Aug 25 17:19 SettingupDrive_GSPGC.ipynb
-rw------- 1 root root 81691 Nov 26 06:47 'Upload data to colab from google drive'

df2 = pd.read_csv('/content/gdrive/My Drive/Colab Notebooks/DOLPHINOFFALLN.csv')

print(df2.head(2))

TIP: We can Install new libraries in python environment inline using below command
Note that appending ! before pip command

!pip install matplotlib

Requirement already satisfied: matplotlib in /usr/local/lib/python3.6/dist-packages (3.0.2)
Requirement already satisfied: numpy>=1.10.0 in /usr/local/lib/python3.6/dist-packages (from matplotlib) (1.14.6)
Requirement already satisfied: cycler>=0.10 in /usr/local/lib/python3.6/dist-packages (from matplotlib) (0.10.0)
Requirement already satisfied: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.1 in /usr/local/lib/python3.6/dist-packages (from matplotlib) (2.3.1)
Requirement already satisfied: kiwisolver>=1.0.1 in /usr/local/lib/python3.6/dist-packages (from matplotlib) (1.0.1)
Requirement already satisfied: python-dateutil>=2.1 in /usr/local/lib/python3.6/dist-packages (from matplotlib) (2.5.3)
Requirement already satisfied: six in /usr/local/lib/python3.6/dist-packages (from cycler>=0.10->matplotlib) (1.11.0)
Requirement already satisfied: setuptools in /usr/local/lib/python3.6/dist-packages (from kiwisolver>=1.0.1->matplotlib) (40.8.0)

References:

https://www.kdnuggets.com/2019/01/more-google-colab-environment-management-tips.html

https://www.kdnuggets.com/2018/02/essential-google-colaboratory-tips-tricks.html

Full code implementation:

Getting started with google laboratory for running deep learning applications

2018-09-08T22:36:00.001+04:00

What is Google Colab:

We all know that deep learning algorithms improve the accuracy of AI applications to great extent. But this accuracy comes with requiring heavy computational processing units such as GPU for developing deep learning models. Many of the machine learning developers cannot afford GPU as they are very costly and find this as a roadblock for learning and developing Deep learning applications. To help the AI, machine learning developers Google has released a free cloud based service Google Colaboratory - Jupyter notebook environment with free GPU processing capabilities with no strings attached for using this service. It is a ready to use service which requires no set at all.

Any AI developers can use this free service to develop deep learning applications using popular AI libraries like Tensorflow, Pytorch, Keras, etc.

Setting up colab:

Go to google drive → new item → More → colaboratory

This opens up a python Jupyter notebook in browser.

By default, the Jupyter notebook runs on Python 2.7 version and CPU processor. We may change the python version to Python 3.6 and processing capability to GPU by changing the settings as shown below:

Go to Runtime → Change runtime type

This opens up a Notebook settings pop-up where we can change Runtime Type to Python 3.6 and processing Hardware to GPU.

Bingo, your python environment with the processing power of GPU is ready use.

Important things to remember:

The supported browsers are Chrome and Firefox
Currently only Python is supported
We can you use upto 12 hours of processing time in one go

Let’s check if our newly created Jupyter notebook works perfectly. Run below commands and see if we are getting expected results.

By default most frequently used python libraries such as Numpy, Pandas, scipy, Sklearn, Matplotlib etc are pre-installed when we create a notebook. Below we can see plotting

Running Machine Learning example:

Below Python Notebook code shows us an example of multi layer neural network using python and sklearn library.

Full code implementation:

In the next post we see how to import data into Colab environment

Getting Started with R

2017-12-02T18:17:00.000+04:00

In this post we get familiar with R Studio and basic syntax of R programming language

Rstudio OverView

we have 4 panes
1) script pane - to write and save the programing script
2) Console pane - where all the code will get executed
3) Environment/history pane - displays all the variables created,functions
used with in the current session
4) Helper pane - contains multiple tabs to install/display pacakges,
view visualization plots,
locate files within the workspace

In [1]:

help(mean)

getting and setting workspace

In [2]:

# to display current working directory use getwd() function
getwd()

'C:/Users/Suresh/mlclassscripts'

In [ ]:

# to set up workspace or working directory use setwd() function
#syntax is shown below
setwd("path")

In [6]:

setwd("C:\\Suresh\\R&D\\Projects\\ML classroom training\\sessions")
setwd("C:/Suresh/R&D/Projects/ML classroom training/sessions")

getting help in R

To get help within R environment, we use help() function to get the
documentation
for any of the functions/packages available within R environment.
To see the arguments required for a function, we use args() function.
to see the example of a function, example() function is used.

In [ ]:

help("stats")
help("mean")
args("mean")
example("mean")

#getting help documentation for a package
help(package="caret")

online help for R programming

We can get online help on available packages in R from official website of R-Cran
https://cran.r-project.org/web/views/

We can also get online support for our day to day activities from below websites:
https://stackoverflow.com/
https://stats.stackexchange.com

Installing Packages

In [ ]:

#install pacakges in R can be done in two ways,
#1) using install.packages() function and from the bottom right pane of Rstudio
install.packages("randomForest")

#loading of installed or downloaded packages can be done using library() function.  
#Note that we can only load the package if
# we have installed the package already within our R environment
library(cluster)

In [ ]:

#below code to first verify if the library is installed in the R environment, 
#if it is not available
# then the package will get installed.
if(!library(cluster)){
install.pacakges("cluster")
}

basic operations in R

In [ ]:

# Adding two numericals
1+1

#multiplying two numericals
10*2

#dividing two numericals
10/2

#applying modulus operation on two numericals
10%%2

printing results to R console

In [ ]:

#printing the data on the console
print(10*2)

print("data science")

print(pi^2)

Variable declaration and assignment in R

variable assignment: In the below example, we are creating variable named z:

In [8]:

z <-  100

we use left arrow or = symbol for variable assignment. Its always good
practice to use left arrow for assignment.

In [9]:

z = 10.009
z <- 10.009

Loading existing or default datasets available in R environment

we can access default datasets avaiable in R using data() function.
data() function will displays all the avaiable datasets within R.

In [ ]:

data()

In order to load a specific dataset into R, we need to give the dataset name as argument to the data() function

In [ ]:

data(AirPassengers)

Viewing data of R objects

To view first 5 records of a R object (ex:dataframe), we use head() function.
head() function expects the data object as argument and prints the first 5 records on the R console.

In [ ]:

head(AirPassengers)

to view all the records in a nice tabular view

In [ ]:

View(AirPassengers)

Getting the description and structure of R object

use str function to see the descriptions of the data object,

In [ ]:

str(AirPassengers)

information retrieval document search using vector space model in R

2017-11-04T17:26:00.000+04:00

Introduction:

In this post, we learn about building a basic search engine or document retrieval system using Vector space model. This use case is widely used in information retrieval systems. Given a set of documents and search term(s)/query we need to retrieve relevant documents that are similar to the search query.

Problem statement:

The problem statement explained above is represented as in below image.

Document retrieval system

Before we get into building the search engine, we will learn briefly about different concepts we use in this post:

Vector Space Model:

A vector space model is an algebraic model, involving two steps, in first step we represent the text documents into vector of words and in second step we transform to numerical format so that we can apply any text mining techniques such as information retrieval, information extraction,information filtering etc.

Let us understand with an example. consider below statements and a query term. The statements are referred as documents hereafter.

Document 1: Cat runs behind rat
Document 2: Dog runs behind cat
Query: rat

Document vectors representation:

In this step includes breaking each document into words, applying preprocessing steps such as removing stopwords, punctuations, special characters etc. After preprocessing the documents we represent them as vectors of words.
Below is a sample representation of the document vectors.

Document 1: (cat, runs, behind, rat)
Document 2: (Dog, runs, behind, cat)
Query: (rat)

the relevant document to Query = greater of (similarity score between (Document1, Query), similarity score between (Document2, Query)

Next step is to represent the above created vectors of terms to numerical format known as term document matrix.

Term Document Matrix:

A term document matrix is a way of representing documents vectors in a matrix format in which each row represents term vectors across all the documents and columns represent document vectors across all the terms. The cell values frequency counts of each term in corresponding document. If a term is present in a document, then the corresponding cell value contains 1 else if the term is not present in the document then the cell value contains 0.

After creating the term document matrix, we will calculate term weights for all the terms in the matrix across all the documents. It is also important to calculate the term weightings because we need to find out terms which uniquely define a document.

We should note that a word which occurs in most of the documents might not contribute to represent the document relevance whereas less frequently occurred terms might define document relevance. This can be achieved using a method known as term frequency - inverse document frequency (tf-idf), which gives higher weights to the terms which occurs more in a document but rarely occurs in all other documents, lower weights to the terms which commonly occurs within and across all the documents.

Tf-idf = tf X idf
tf = term frequency is the number of times a term occurs in a document
idf = inverse of the document frequency, given as below
idf = log(N/df), where df is the document frequency-number of documents containing a term

total number of documents

term document matrix

inverse document frequency

Note: idf is calculated using logarithm of inverse fraction between document count and document frequency

tf-idf calculation

Note: Tf-idf weightage is calculated using tf X idf

Note, there are many variations in the way we calculate the term-frequency(tf) and inverse document frequency (idf), in this post we have seen one variation. Below images show as the other recommended variations of tf and idf, taken from wiki.

term frequency variations

inverse document frequency variations

Similarity Measures: cosine similarity

Mathematically, closeness between two vectors is calculated by calculating the cosine angle between two vectors. In similar lines, we can calculate cosine angle between each document vector and the query vector to find its closeness. To find relevant document to the query term , we may calculate the similarity score between each document vector and the query term vector by applying cosine similarity . Finally, whichever documents having high similarity scores will be considered as relevant documents to the query term.

When we plot the term document matrix, each document vector represents a point in the vector space. In the below example query, Document 1 and Document 2 represent 3 points in the vector space. We can now compare the query with each of the document by calculating the cosine angle between them.

cosine similarity

Apart from cosine similarity, we have other variants for calculating the similarity scores and are shown below:

Jaccard distance
Kullback-Leibler divergence
Euclidean distance

Now that we have learnt the important concepts required for implementing our problem statement, we now look at the data which will be used in this post and its implementation in R programming language.

Data description:

For this post, we use 9 text files containing news articles and a query file containing search queries. Our task is to find top-3 news articles relevant to each of the query in the queries files.
The dataset which we will be using is uploaded to GitHub and is located at below location:
https://github.com/sureshgorakala/machinelearning/tree/master/data

The news articles data is available in txt files as shown in below image:

Below is the snippet of news article in the first txt file baract_hussein_obama.txt,
“Barack Hussein Obama II (US Listeni/bəˈrɑːk huːˈseɪn oʊˈbɑːmə/;[1][2] born August 4, 1961) is the 44th and current President of the United States. He is the first African American to hold the office and the first president born outside the continental United States. Born in Honolulu, Hawaii, Obama is a graduate of Columbia University and Harvard Law School, where he was president of the Harvard Law Review. He was a community organizer in Chicago before earning his law degree. He worked as a civil rights attorney and taught constitutional law at the University of Chicago Law School between 1992 and 2004. While serving three terms representing the 13th District in the Illinois Senate from 1997 to 2004, he ran unsuccessfully in the Democratic primary for the United States House of Representatives in 2000 against incumbent Bobby Rush ….”
Below are the sample queries to which we will extract relevant documents, available in query.txt file, is shown below:
“largest world economy
barack obama
united state president
donald trump and united state
donald trump and barack obama
current President of the United States”
our task is to create a system in which for each of the query terms retrieve top-3 relevant documents.

High level system design:

In this section we show the high-level design implementation. The implementation steps are as follows:

Load documents and search queries into the R programming environment as list objects.
Preprocess the data by creating a corpus object with all the documents and query terms, removing stop words, punctuations using tm package.

high level information retrieval system

Creating a term document matrix with tf-idf weight setting available in TermDocumentMatrix() method.
Separate the term document matrix into two parts- one containing all the documents with term weights and other containing all the queries with term weights.
Now calculate cosine similarity between each document and each query.
For each query sort the cosine similarity scores for all the documents and take top-3 documents having high scores.

Full code implementation:

apply lapply rapply sapply functions in R

2016-03-18T21:42:00.000+04:00

As part of Data Science with R, this is third tutorial after basic data types,control structures in r.

One of the issues with for loop is its memory consumption and its slowness in executing a repetitive task at hand. Often dealing with large data and iterating it, for loop is not advised. R provides many few alternatives to be applied on vectors for looping operations. In this section, we deal with apply function and its variants:

?apply

Datasets for apply family tutorial

 For understanding the apply functions in R we use,the data from 1974 Motor Trend
US magazine which comprises fuel consumption and 10 aspects of automobile design and
performance for 32 automobiles (1973–74 models).
 
data("mtcars")
head(mtcars)
                   mpg cyl disp  hp drat    wt  qsec vs am gear carb
Mazda RX4         21.0   6  160 110 3.90 2.620 16.46  0  1    4    4
Mazda RX4 Wag     21.0   6  160 110 3.90 2.875 17.02  0  1    4    4
Datsun 710        22.8   4  108  93 3.85 2.320 18.61  1  1    4    1
Hornet 4 Drive    21.4   6  258 110 3.08 3.215 19.44  1  0    3    1
Hornet Sportabout 18.7   8  360 175 3.15 3.440 17.02  0  0    3    2
Valiant           18.1   6  225 105 2.76 3.460 20.22  1  0    3    1
 

Reynolds (1994) describes a small part of a study of the long-term temperature dynamics
of beaver Castor canadensis in north-central Wisconsin. Body temperature was measured by
telemetry every 10 minutes for four females, but data from a one period of less than a 
day for each of two animals is used there. 

data(beavers)
head(t(beaver1)[1:4,1:10])
        [,1]   [,2]   [,3]   [,4]   [,5]   [,6]   [,7]   [,8]    [,9]   [,10]
day   346.00 346.00 346.00 346.00 346.00 346.00 346.00 346.00  346.00  346.00
time  840.00 850.00 900.00 910.00 920.00 930.00 940.00 950.00 1000.00 1010.00
temp   36.33  36.34  36.35  36.42  36.55  36.69  36.71  36.75   36.81   36.88
activ   0.00   0.00   0.00   0.00   0.00   0.00   0.00   0.00    0.00    0.00

apply():
apply() function is the base function. We will learn how to apply family functions by trying out the code. apply() function takes 3 arguments:

data matrix
row/column operation, - 1 for row wise operation, 2 for column wise operation
function to be applied on the data.

 
when 1 is passed as second parameter, the function max is applied row wise and gives
us the result. In the below example, row wise maximum value is calculated.Since we 
have four types of attributes we got 4 results.
 
apply(t(beaver1),1,max) 
    day    time    temp   activ 
 347.00 2350.00   37.53    1.00 

 
When 2 is passed as second  parameter the function  mean is applied column wise.
In the below example mean function is applied on each column and mean for each 
column is calculated. Hence  we can see results for each column.
 
apply(mtcars,2,mean) 
       mpg        cyl       disp         hp       drat         wt       qsec         vs         am       gear       carb 
 20.090625   6.187500 230.721875 146.687500   3.596563   3.217250  17.848750   0.437500   0.406250   3.687500   2.812500 
 
We can also pass custom function instead of default functions. For example in 
the below example let us divide each column element with modulus of 10.
For this we use a custom function which takes each element from each column and
apply the modulus operation.
 
head(apply(mtcars,2,function(x) x%%10))
                  mpg cyl disp hp drat    wt qsec vs am gear carb
Mazda RX4         1.0   6    0  0 3.90 2.620 6.46  0  1    4    4
Mazda RX4 Wag     1.0   6    0  0 3.90 2.875 7.02  0  1    4    4
Datsun 710        2.8   4    8  3 3.85 2.320 8.61  1  1    4    1
Hornet 4 Drive    1.4   6    8  0 3.08 3.215 9.44  1  0    3    1
Hornet Sportabout 8.7   8    0  5 3.15 3.440 7.02  0  0    3    2
Valiant           8.1   6    5  5 2.76 3.460 0.22  1  0    3    1

lapply():
lapply function is applied for operations on list objects and returns a list object of same length of original set.
lapply function in R, returns a list of the same length as input list object, each element of which is the result of applying FUN to the corresponding element of list.

 #create a list with 2 elements 
l = (a=1:10,b=11:20)  # the mean of the value in each element
lapply(l, mean)
$a
[1] 5.5
$b
[1] 15.5
class(lapply(l, mean))
[1] "list
  # the sum of the values in each element 
lapply(l, sum)
$a
[1] 55

$b
[1] 155

sapply():
sapply is wrapper class to lapply with difference being it returns vector or matrix instead of list object.

 
 # create a list with 2 elements 
 l = (a=1:10,b=11:20)  # mean of values using sapply 
sapply(l, mean)
   a    b 
 5.5 15.5

tapply():
tapply() is a very powerful function that lets you break a vector into pieces and then apply some function to each of the pieces. In the below code, first each of mpg in mtcars data is grouped by cylinder type and then mean() function is calculated.

str(mtcars$cyl)
 num [1:32] 6 6 4 6 8 6 8 4 4 6 ...
levels(as.factor(mtcars$cyl))
[1] "4" "6" "8"

In the dataset we have 3 types of cylinders and now we want to see the average mpg
for each cylinder type.

tapply(mtcars$mpg,mtcars$cyl,mean)
       4        6        8 
26.66364 19.74286 15.10000 

In the output above we see that the average mpg for 4 cylinder engine 
is 26.664, 6-cyinder engine is 19.74 and 8-cylinder engine is 15.10

by():
by works similar to group by function in SQL, applied to factors, where in we may apply operations on individual results set. In the below example, we apply colMeans() function to all the observations on iris dataset grouped by Species.

data(iris) 
'data.frame': 150 obs. of  5 variables:
 $ Sepal.Length: num  5.1 4.9 4.7 4.6 5 5.4 4.6 5 4.4 4.9 ...
 $ Sepal.Width : num  3.5 3 3.2 3.1 3.6 3.9 3.4 3.4 2.9 3.1 ...
 $ Petal.Length: num  1.4 1.4 1.3 1.5 1.4 1.7 1.4 1.5 1.4 1.5 ...
 $ Petal.Width : num  0.2 0.2 0.2 0.2 0.2 0.4 0.3 0.2 0.2 0.1 ...
 $ Species     : Factor w/ 3 levels "setosa","versicolor",..: 1 1 1 1 1 1 1 1 1 1 ...

by(iris[,1:4],iris$Species,colMeans)
iris$Species: setosa
Sepal.Length  Sepal.Width Petal.Length  Petal.Width 
       5.006        3.428        1.462        0.246 
------------------------------------------------------------------------------------ 
iris$Species: versicolor
Sepal.Length  Sepal.Width Petal.Length  Petal.Width 
       5.936        2.770        4.260        1.326 
------------------------------------------------------------------------------------ 
iris$Species: virginica
Sepal.Length  Sepal.Width Petal.Length  Petal.Width 
       6.588        2.974        5.552        2.026

rapply():
rapply() is a recursive version of lapply.
rapply() applies a function recursively on each element of the list with 2 modes for "how" parameter. If how = "replace", each element of the list which is not itself a list and has a class included in classes is replaced by the result of applying f to the element.If the mode is how = "list" or how = "unlist", the list is copied, all non-list elements which have a class included in classes are replaced by the result of applying f to the element and all others are replaced by deflt. Finally, if how = "unlist", unlist(recursive = TRUE) is called on the result.
'

l2 = list(a = 1:10, b = 11:20,c=c('d','a','t','a'))
l2
$a
 [1]  1  2  3  4  5  6  7  8  9 10

$b
 [1] 11 12 13 14 15 16 17 18 19 20

$c
[1] "d" "a" "t" "a"

rapply(l2, mean, how = "list", classes = "integer")
$a
[1] 5.5

$b
[1] 15.5

$c
NULL

rapply(l2, mean, how = "unlist", classes = "integer")
 a    b 
 5.5 15.5 
 
rapply(l2, mean, how = "replace", classes = "integer")
$a
[1] 5.5

$b
[1] 15.5

$c
[1] "d" "a" "t" "a"

mapply():
mapply is a multivariate version of sapply. By R definition, mapply is a multivariate version of sapply. mapply applies FUN to the first elements of each ... argument, the second elements, the third elements, and so on. Arguments are recycled if necessary. Its purpose is to be able to vectorize arguments to a function that is not usually accepting vectors as arguments. In short, mapply applies a Function to Multiple List or multiple Vector Arguments. In the below example word function is applied to vector argument LETTERS. '

word = function(C, k) paste(rep.int(C, k), collapse = "")
utils::str(mapply(word, LETTERS[1:6], 6:1, SIMPLIFY = FALSE))
List of 6
 $ A: chr "AAAAAA"
 $ B: chr "BBBBB"
 $ C: chr "CCCC"
 $ D: chr "DDD"
 $ E: chr "EE"
 $ F: chr "F"

Control Structures Loops in R

2016-02-27T13:51:00.000+04:00

As part of Data Science tutorial Series in my previous post I posted on basic data types in R. I have kept the tutorial very simple so that beginners of R programming may takeoff immediately.
Please find the online R editor at the end of the post so that you can execute the code on the page itself.
In this section we learn about control structures loops used in R. Control strcutures in R contains conditionals, loop statements like any other programming languages.

Loops are very important and forms backbone to any programming languages.Before we get into the control structures in R, just type as below in Rstudio:

 ?control

If else statement:

#See the code syntax below for if else statement 
if(x>1){
 print("x is greater than 1")
 }else{
  print("x is less than 1")
  } 

#See the code below for nested if else statement

 x=10
  x=10
 if(x>1 & x<7){
     print("x is between 1 and 7")}else if(x>8 & x< 15){
         print("x is between 8 and 15")
     }

[1] "x is between 8 and 15"

For loops:
As we know for loops are used for iterating items

 #Below code shows for  loop implementation
x = c(1,2,3,4,5)
 for(i in 1:5){
     print(x[i])
 }
[1] 1
[1] 2
[1] 3
[1] 4
[1] 5

While loop :

 #Below code shows while loop in R
x = 2.987
while(x <= 4.987) { 
     x = x + 0.987
     print(c(x,x-2,x-1)) 
 }
[1] 3.974 1.974 2.974
[1] 4.961 2.961 3.961
[1] 5.948 3.948 4.948

Repeat Loop:
The repeat loop is an infinite loop and used in association with a break statement.

 #Below code shows repeat loop:
a = 1
 repeat { print(a) a = a+1 if(a > 4) break }
[1] 1
[1] 2
[1] 3
[1] 4

Break statement:
A break statement is used in a loop to stop the iterations and flow the control outside of the loop.

 #Below code shows break statement:
x = 1:10 
 for (i in x){ 
     if (i == 2){ 
         break 
     }
     print(i)
 }
[1] 1

Next statement:
Next statement enables to skip the current iteration of a loop without terminating it.

 #Below code shows next statement 
x = 1: 4 
 for (i in x) { 
     if (i == 2){ 
         next}
     print(i)
 }
[1] 1
[1] 3
[1] 4

Creating a function in R:
function() is a built-in R function whose job is to create functions. In the below example function() takes one parameter x, executes a for loop logic.
The function object thus created using function() is assigned to a variable ('words.names'). Now this created function will be called using the variable 'word.names'

 #Below code shows us, how a function is created in R:

Syntax: 
function_name = function(parameters,..){ code}
 
words = c("R", "datascience", "machinelearning","algorithms","AI") 
words.names = function(x) {
     for(name in x){ 
         print(name) 
     }
} 
#Calling the function
 words.names(words)
[1] "R"
[1] "datascience"
[1] "machinelearning"
[1] "algorithms"
[1] "AI"

Hands on exercise of what we have learnt so far

We create a data frame DF, run for loop, ifelse in a function and call the function
#create 3 vectors name,age,salary
name = c("David","John","Mathew")
age = c(30,40,50)
salary = c(30000,120000,55000)
#create a data frame DF by combining the 3 vectors using cbind() function
DF = data.frame(cbind(name,age,salary))
#display DF
DF
    name age salary
1  David  30  30000
2   John  40 120000
3 Mathew  50  55000
#dimensions of DF
 dim(DF)
[1] 3 3
 
#write a function which displays the salaried person name
findHighSalary = function(df){
     Maxsal = 0
     empname = ""
     for(i in 1:nrow(DF)){
         tmpsal = as.numeric(DF[i,3] )
         if(tmpsal > Maxsal){
             Maxsal = tmpsal
             empname = DF[i,1]
         }
     }
     return(as.character(empname))
 }
#calling the function
findHighSalary(DF)
[1] "Mathew"

Principal Component Analysis using R

2016-02-27T12:18:00.000+04:00

Curse of Dimensionality:
One of the most commonly faced problems while dealing with data analytics problem such as recommendation engines, text analytics is high-dimensional and sparse data. At many times, we face a situation where we have a large set of features and fewer data points, or we have data with very high feature vectors. In such scenarios, fitting a model to the dataset, results in lower predictive power of the model. This scenario is often termed as the curse of dimensionality. In general, adding more data points or decreasing the feature space, also known as dimensionality reduction, often reduces the effects of the curse of dimensionality.
In this blog, we will discuss about principal component analysis, a popular dimensionality reduction technique. PCA is a useful statistical method that has found application in a variety of fields and is a common technique for finding patterns in data of high dimension.

Principal component analysis:

Consider below scenario:
The data, we want to work with, is in the form of a matrix A of mXn dimension, shown as below, where Ai,j represents the value of the i-th observation of the j-th variable.

Thus the N members of the matrix can be identified with the M rows, each variable corresponding to N-dimensional vectors. If N is very large it is often desirable to reduce the number of variables to a smaller number of variables, say k variables as in the image below, while losing as little information as possible.

Mathematically spoken, PCA is a linear orthogonal transformation that transforms the data to a new coordinate system such that the greatest variance by any projection of the data comes to lie on the first coordinate (called the first principal component), the second greatest variance on the second coordinate, and so on.

The algorithm when applied linearly transforms m-dimensional input space to n-dimensional (n < m) output space, with the objective to minimize the amount of information/variance lost by discarding (m-n) dimensions. PCA allows us to discard the variables/features that have less variance.
Technically speaking, PCA uses orthogonal projection of highly correlated variables to a set of values of linearly uncorrelated variables called principal components. The number of principal components is less than or equal to the number of original variables. This linear transformation is defined in such a way that the first principal component has the largest possible variance. It accounts for as much of the variability in the data as possible by considering highly correlated features. Each succeeding component in turn has the highest variance using the features that are less correlated with the first principal component and that are orthogonal to the preceding component.

In the above image, u1 & u2 are principal components wherein u1 accounts for highest variance in the dataset and u2 accounts for next highest variance and is orthogonal to u1.

PCA implementation in R:

For today’s post we use crimtab dataset available in R. Data of 3000 male criminals over 20 years old undergoing their sentences in the chief prisons of England and Wales. The 42 row names ("9.4", 9.5" ...) correspond to midpoints of intervals of finger lengths whereas the 22 column names ("142.24", "144.78"...) correspond to (body) heights of 3000 criminals, see also below.

head(crimtab)
    142.24 144.78 147.32 149.86 152.4 154.94 157.48 160.02 162.56 165.1 167.64 170.18 172.72 175.26 177.8 180.34
9.4      0      0      0      0     0      0      0      0      0     0      0      0      0      0     0      0
9.5      0      0      0      0     0      1      0      0      0     0      0      0      0      0     0      0
9.6      0      0      0      0     0      0      0      0      0     0      0      0      0      0     0      0
9.7      0      0      0      0     0      0      0      0      0     0      0      0      0      0     0      0
9.8      0      0      0      0     0      0      1      0      0     0      0      0      0      0     0      0
9.9      0      0      1      0     1      0      1      0      0     0      0      0      0      0     0      0
    182.88 185.42 187.96 190.5 193.04 195.58
9.4      0      0      0     0      0      0
9.5      0      0      0     0      0      0
9.6      0      0      0     0      0      0
9.7      0      0      0     0      0      0
9.8      0      0      0     0      0      0
9.9      0      0      0     0      0      0
 dim(crimtab)
[1] 42 22
str(crimtab)
 'table' int [1:42, 1:22] 0 0 0 0 0 0 1 0 0 0 ...
 - attr(*, "dimnames")=List of 2
  ..$ : chr [1:42] "9.4" "9.5" "9.6" "9.7" ...
  ..$ : chr [1:22] "142.24" "144.78" "147.32" "149.86" ...

sum(crimtab)
[1] 3000

colnames(crimtab)
 [1] "142.24" "144.78" "147.32" "149.86" "152.4"  "154.94" "157.48" "160.02" "162.56" "165.1"  "167.64" "170.18" "172.72" "175.26" "177.8"  "180.34"
[17] "182.88" "185.42" "187.96" "190.5"  "193.04" "195.58"

let us use apply() to the crimtab dataset row wise to calculate the variance to see how each variable is varying.

apply(crimtab,2,var)

We observe that column “165.1” contains maximum variance in the data. Applying PCA using prcomp().

pca =prcomp(crimtab)

pca


Note: the resultant components of pca object from the above code corresponds to the standard deviations and Rotation. From the above standard deviations we can observe that the 1st PCA explained most of the variation, followed by other pcas’.  Rotation contains the principal component loadings matrix values which explains /proportion of each variable along each principal component.

Let’s plot all the principal components and see how the variance is accounted with each component.

par(mar = rep(2, 4))
plot(pca)

Clearly the first principal component accounts for maximum information.
Let us interpret the results of pca using biplot graph. Biplot is used to show the proportions of each variable along the two principal components.

#below code changes the directions of the biplot, if we donot include the below two lines the plot will be mirror image to the below one.
pca$rotation=-pca$rotation
pca$x=-pca$x
biplot (pca , scale =0)

The output of the preceding code is as follows:

In the preceding image, known as a biplot, we can see the two principal components (PC1 and PC2) of the crimtab dataset. The red arrows represent the loading vectors, which represent how the feature space varies along the principal component vectors.
From the plot, we can see that the first principal component vector, PC1, more or less places equal weight on three features: 165.1, 167.64, and 170.18. This means that these three features are more correlated with each other than the 160.02 and 162.56 features.
In the second principal component, PC2 places more weight on 160.02, 162.56 than the 3 features, "165.1, 167.64, and 170.18" which are less correlated with them.
Complete Code for PCA implementation in R:

So by now we understood how to run the PCA, and how to interpret the principal components, where do we go from here? How do we apply the reduced variable dataset? In our next post we shall answer the above questions.

Basic Data Types in R

2016-02-16T19:00:00.001+04:00

As part of tutorial series on Data Science with R from Data Perspective, this first tutorial introduces the very basics of R programming language about basic data types in R.

What we learn:
After the end of the chapter, you are provided with R console so that you can practice what you have learnt in this chapter.

R assignment operator

x = 'welcome to R programming' # assigning string literal to variable x 
x
[1] "welcome to R programming"
typeof(x) #to check the data type of the variable x
[1] "character"

Numeric

Numeric data represents decimal data.

x = 1.5 #assigning decimal value1.5 to x
x
[1] 1.5

To check the data type we use class() function:

class(x)
[1] "numeric"

To check if the variable “x” is of numerical or not, we use

is.numeric(x)
[1] TRUE

To convert any compatible data into numeric, we use:

 x = '1' #assigning value 1 to variable x
 class(x)
[1] "character"
 x = as.numeric(x)
[1] 1
 class(x)
[1] "numeric"

Note: if we try to convert a string literal to numeric data type we get the following result.

x= 'welcome to R programming'
as.numeric(x)
[1] NA
Warning message:
NAs introduced by coercion

Integer

We use as.integer() function to convert into integers. This converts numeric value to integer values.

x = 1.34
 [1] 1.34
class(x)
[1] "numeric"
y = as.integer(x)
class(y)
[1] "integer"
y
[1] 1

Note: to check if the value is integer or not we use is.integer() function.
In the below example ‘y’ is numerical or decimal value whereas x is integer.

 is.integer(y)
[1] TRUE
 is.integer(x)
[1] FALSE

Complex:

Complex data types are shown as below, though we use it very less in our day to day data analysis:

c = 3.5+4i
[1] 3.5+4i
is.complex(c)
[1] TRUE
class(c)
[1] "complex"

Logical

Logical data type is one of the frequently used data type usually used for comparing two values. Values a logical data type takes is TRUE or FALSE.

logical = T
logical
[1] TRUE
l = FALSE
l
[1] FALSE

Character

String literals or string values are stored as Character objects in R.

str = "R Programming"
str
[1] "R Programming"
class(str)
[1] "character"
is.character(str)
[1] TRUE

We can convert other data types to character data type using as.character() function.

x = as.character(1)
x
[1] "1"
class(x)
[1] "character"

Note: There are a variety of operations that can be applied on characters such as substrings, finding lengths; etc will be dealt as when appropriate.
So far we learnt about the basic data types in R, let’s get into a bit complex data types.

Vector

How do we hold collection of same data types? We come across this requirement very frequently. We have vector data type to solve this problem.
Consider a numerical vector below:

num_vec = c(1,2,3,4,5)
num_vec
[1] 1 2 3 4 5
class(num_vec)
[1] "numeric"

We can apply many operations on the vector variables such as length, accessing values or members of the vector variable.
Length of the vector can be found using length() function.

length(num_vec)
[1] 5

We access each element or member of the vector num_vec using its indexes starting from
In the below example we can access the members at 1st,2nd,3rd positons.

num_vec[1]
[1] 1
num_vec[2]
[1] 2
num_vec[3]
[1] 3

Similarly string vectors, logical vectors, integer vectors can be created.

char_vec = c("A", "Course","On","Data science","R rprogramming")
char_vec
[1] "A"  "Course" "On" "Data science" "R rprogramming"
length(char_vec)
[1] 5
char_vec[1]
[1] "A"
char_vec[2]
[1] "Course"
char_vec[4]
[1] "Data science"

Matrix

Matrix data type is used when we want to represent the data as collection of numerical values in mXn, m by n, dimensions. Matrices are used mostly when dealing with mathematical equations, machine learning, text mining algorithms.
Now how do we create a matrix?

m = matrix(c(1,2,3,6,7,8),nrow = 2,ncol = 3)
m
     [,1] [,2] [,3]
[1,]    1    3    7
[2,]    2C    6    8
class(m)
[1] "matrix"

Knowing the dimension of the matrix is:

dim(m)
[1] 2 3

How do we access elements of matrix m:

#accessing individual elements are done using the indexes shown as below. In the below example we are accessing 1st, 2nd, 6th element of matrix m.
m[1]
[1] 1
m[2]
[1] 2
m[6]
[1] 8
m[2,3] #  here we accessing 2nd row 3rd column element.
[1] 8
# accessing all elements of rows of the matrix m shown below. 
m[1,]
[1] 1 3 7
m[2,]
[1] 2 6 8
#accessing all elements of each column
m[,1]
[1] 1 2
m[,2]
[1] 3 6
m[,3]
[1] 7 8

What happens when we add different data types to a vector?

v = c("a","b",1,2,3,T)
v
[1] "a"    "b"    "1"    "2"    "3"    "TRUE"
class(v)
[1] "character"
v[6]
[1] "TRUE"
class(v[6])
[1] "character"

What happened in the above example is, R coerced all different data types into a single data type of character type to maintain the condition of single data type.

List

What if we want to handle different data types in a single object?
List data type helps us in storing elements of different data types in a single object.
We create list objects using list() function.
In the below example I have created a list object “list_exp” with 6 different elements of character, numeric and logical data types.

list_exp = list("r programming","data perspective",12345,67890,TRUE,F)
list_exp
[[1]]
[1] "r programming"
[[2]]
[1] "data perspective"
[[3]]
[1] 12345
[[4]]
[1] 67890
[[5]]
[1] TRUE
[[6]]
[1] FALSE

Using str() function, we can know the structure of the list object, i.e. the internal structure of the list objects can be known. This is one of the very important functions which we use in our day to day analysis.
In the below example we can see a list of 6 elements of character, numerical and logical data types.

str(list_exp)
List of 6
 $ : chr "r programming"
 $ : chr "data perspective"
 $ : num 12345
 $ : num 67890
 $ : logi TRUE
 $ : logi FALSE
#accessing the  data type of list_exp
class(list_exp)
[1] "list"
length(list_exp)
[1] 6
list_exp[1]
[[1]]
[1] "r programming"
#accessing the list elements using indexing.
list_exp[[1]]
[1] "r programming"
list_exp[[6]]
[1] FALSE
list_exp[[7]] # when we try accessing not existing elements we get the below error.
Error in list_exp[[7]] : subscript out of bounds
# finding the class of individual list element
class(list_exp[[6]])
[1] "logical"
class(list_exp[[3]])
[1] "numeric"
class(list_exp[[1]])
[1] "character"

Data Frame:

Most of us would be from a bit of SQL background and we would be very much comfortable in handling data in the form of SQL table because of the functionalities which a SQL table offers while working the data.
How would it be if we have such data type object available in R which can be used to store the data and manipulate data in very easy, efficient and convenient way?
R offers a data frame data type object. It is another way that information is stored as data frames. We can treat a data frame similar to a SQL table.
How do we create a data frame?

#creating a data frame
data_frame = data.frame(first=c(1,2,3,4),second=c("a","b","c","d"))
data_frame
  first second
     1      a
     2      b
     3      c
     4      d
#accessing  the data type of the object
class(data_frame)
[1] "data.frame"
#finding out the row count of data_frame using nrow()
nrow(data_frame)
[1] 4
#finding out the column count of data_frame using ncol()
ncol(data_frame)
[1] 2
#finding out the dimensions of data_frame using dim()
dim(data_frame)
[1] 4 2
#finding the structure of the data frame using str()
str(data_frame)
'data.frame': 4 obs. of  2 variables:
 $ first : num  1 2 3 4
 $ second: Factor w/ 4 levels "a","b","c","d": 1 2 3 4
#accessing the entire row of data frame using row index number. Observe below that if we use data_frame[1,] without specifying the column number it means that we want to access all the columns of row 1.
data_frame[1,]
  first second
1     1      a
#similarly to access only 1st column values without row information use data_frame[,1] 
data_frame[,1]
[1] 1 2 3 4
#accessing the row names of the data frame.
rownames(data_frame)
[1] "1" "2" "3" "4"
#accessing the column names of the data frame
colnames(data_frame)
[1] "first"  "second"
#column data can accessed using the column names explicitly instead of column indexes
data_frame$first
[1] 1 2 3 4
data_frame$second
[1] a b c d
Levels: a b c d
#accessing individual values using row and column indexes
data_frame[1,1] # accessing first row first column
[1] 1
data_frame[2,2] # accessing second row second column
[1] b
Levels: a b c d
data_frame[3,2]  # accessing third row second column
[1] c
Levels: a b c d
data_frame[3,1] # accessing third row first column
[1] 3

Note: Observe the below data frame:

dt_frame = data.frame(first=c(1,2,3,4,5,6,7),second=c("Big data","Python","R","NLP","machine learning","data science","data perspective"))
dt_frame
  first           second
     1         Big data
     2           Python
     3                R
     4              NLP
     5 machine learning
     6     data science
     7 data perspective

Assume we have a dataset with 1000 rows instead of 6 rows shown in above data frame. If we want to see a sample of data of the data frame, how do we do?
Using head() function.

head(dt_frame)
  first           second
     1         Big data
     2           Python
     3                R
     4              NLP
     5 machine learning
     6     data science

head() function returns us the first six rows of any data frame so that we can have a look of what the data frame is like.
Also we can use tail() function to see the last six rows of the data frame.

tail(dt_frame)
  first           second
     2           Python
     3                R
     4              NLP
     5 machine learning
     6     data science
     7 data perspective

We have View() function to see the values of a data frame in a tabular form.
View(dt_frame)

Try Code

Data Science with R

2015-12-25T05:42:00.002+04:00

As R programming language becoming popular more and more among data science group, industries, researchers, companies embracing R, going forward I will be writing posts on learning Data science using R. The tutorial course will include topics on data types of R, handling data using R, probability theory, Machine Learning, Supervised – unSupervised learning, Data Visualization using R, etc. Before going further, let’s just see some stats and tidbits on data science and R.

"A data scientist is simply someone who is highly adept at studying large amounts of often unorganized/undigested data"

“R programming language is becoming the Magic Wand for Data Scientists”

Why R for Data Science?

Daryl Pregibon, a research scientist at Google said- “R is really important to the point that it’s hard to overvalue it. It allows statisticians to do very intricate and complicated data analysis without knowing the blood and guts of computing systems.”

A brief stats for R popularity

Source: 2015 Analytics/Data Mining Software Poll

“The shortage of data scientists is becoming a serious constraint in some sectors”

David Smith, Chief Community Officer at Revolution Analytics said –
“Investing in R, whether from the point of view of an individual Data Scientist or a company as a whole is always going to pay off because R is always available. If you’ve got a Data Scientist new to an organization, you can always use R. If you’re a company and you’re putting your practice on R, R is always going to be available. And, there’s also an ecosystem of companies built up around R including Revolution Enterprise to help organizations implement R into their machine critical production processes.”

Enough of candies about R, The topics which I cover in the course are:

R Basics
Probability theory in R
Machine Learning in R
Supervised machine learning
Unsupervised machine learning
Advanced Machine Learning in R
Data Visualization in R

See you in the first chapter, meanwhile read about the various data analysis steps involved.

Item Based Collaborative Filtering Recommender Systems in R

2015-11-18T22:39:00.001+04:00

In the series of implementing Recommendation engines, in my previous blog about recommendation system in R, I have explained about implementing user based collaborative filtering approach using R. In this post, I will be explaining about basic implementation of Item based collaborative filtering recommender systems in r.
Intuition:

Item based Collaborative Filtering:
Unlike in user based collaborative filtering discussed previously, in item-based collaborative filtering, we consider set of items rated by the user and computes item similarities with the targeted item. Once similar items are found, and then rating for the new item is predicted by taking weighted average of the user’s rating on these similar items.
let's understand with an example:
As an example: consider below dataset, containing users rating to movies. Let us build an algorithm to recommend movies to CHAN.

Implementing Item based recommender systems, like user based collaborative filtering, requires two steps:

Calculating Item similarities
Predicting the targeted item rating for the targeted User.

Step1: Calculating Item Similarity:
This is a critical step; we calculate the similarity between co-rated items. We use cosine similarity or pearson-similarity to compute the similarity between items. The output for step is similarity matrix between Items.

Code snippet:

#step 1: item-similarity calculation co-rated items are considered and similarity between two items

#are calculated using cosine similarity

library(lsa)

ratings = read.csv("Rating Matrix.csv")

x = ratings[,2:7]

x[is.na(x)] = 0

item_sim = cosine(as.matrix(x))

Step2: Predicting the targeted item rating for the targeted User CHAN.
In this most important step, we first predict the items which the user is not rated by making use of the ratings he has made to previously interacted items and the similarity values calculated in the previous step. First we select item to be predicted, in our case “INCEPTION”, we predict the rating for INCEPTION movie by calculating the weighted sum of ratings made to movies similar to INCEPTION. i.e We take the similarity score for each rated movie by CHAN w.r.t INCEPTION and multiply with the corresponding rating and sum up all the for all the rated movies. This final sum is divided by total sum of similarity scores of rated items w.r.t INCEPTION.

Recommending Top N items:
Once all the non rated movies are predicted we recommend top N movies to CHAN. Code for Item based collaborative filtering in R:

#data input

ratings = read.csv("~Rating Matrix.csv")

"step 1: item-similarity calculation\nco-rated items are considered and similarity between two items\nare calculated using cosine similarity"

library(lsa)

x = ratings[,2:7]

x[is.na(x)] = 0

item_sim = cosine(as.matrix(x))

"Recommending items for chan: since three movies are not rated\nas a first step we have to predict rating value for each movie\nin CHANs case we have to first predict values for Titanic, Inception,Matrix"

rec_itm_for_user = function(userno)

{

#extract all the movies not rated by CHAN

userRatings = ratings[userno,]

non_rated_movies = list()

rated_movies = list()

for(i in 2:ncol(userRatings)){

if(is.na(userRatings[,i]))

{

non_rated_movies = c(non_rated_movies,colnames(userRatings)[i])

}

else

{

rated_movies = c(rated_movies,colnames(userRatings)[i])

}

non_rated_movies = unlist(non_rated_movies)

rated_movies = unlist(rated_movies)

#create weighted similarity for all the rated movies by CHAN

non_rated_pred_score = list()

for(j in 1:length(non_rated_movies)){

temp_sum = 0

df = item_sim[which(rownames(item_sim)==non_rated_movies[j]),]

for(i in 1:length(rated_movies)){

temp_sum = temp_sum+ df[which(names(df)==rated_movies[i])]

}

weight_mat = df*ratings[userno,2:7]

non_rated_pred_score = c(non_rated_pred_score,rowSums(weight_mat,na.rm=T)/temp_sum)

}

pred_rat_mat = as.data.frame(non_rated_pred_score)

names(pred_rat_mat) = non_rated_movies

for(k in 1:ncol(pred_rat_mat)){

ratings[userno,][which(names(ratings[userno,]) == names(pred_rat_mat)[k])] = pred_rat_mat[1,k]

}

return(ratings[userno,])

}

> rec_itm_for_user(7)

  Users  Titanic Batman Inception SuperMan Spiderman   matrix

7  CHAN 3.085298    4.5  2.940811        4         1 3.170034

Calling above function gives the predicted values not previously seen values for movies Titanic, Inception, Matrix. Now we can sort and recommend the top items.
This is all about Collaborative filtering in R, in my upcoming posts I will talk about content based recommender systems in r.

Data Mining Standard Process across Organizations

2015-10-19T05:47:00.000+04:00

Recently I have come across a term, CRISP-DM - a data mining standard. Though this process is not a new one but I felt every analyst should know about commonly used Industry wide process. In this post I will explain about different phases involved in creating a data mining solution.

CRISP-DM, an acronym for Cross Industry Standard Process for Data Mining, is a data mining process model that includes commonly used approaches that data analytics Organizations use to tackle business problems related to Data mining. Polls conducted at one and the same website (KDNuggests) in 2002, 2004, 2007 and 2014 show that it was the leading methodology used by industry data miners who decided to respond to the survey.

CRISP-DM model is a phased approach to tackle a business problem. Different phases involved in the model are defined below:

Use case Identification
Business Understanding
Data Acquisition and Data Understanding
Data Preparation
Exploratory Analysis
Data Modeling
Data Evaluation
Deployment

Let us see each of the phases in detailed way:

Use case Identification:This is the initial phase of CRISP-DM in which a potential business problem is formulated into a Data mining use case. Various levels of brainstorming sessions are conducted between different stakeholders to define the problem statement, its impact on the business and a clear objective of the solution and its timelines.
Audience:

higher management
IT teams – Application team, DBA team
Analytics team – Data Scientist

Business Understanding:This phase focuses on understanding the business flow – how the current system addresses the business problem in consideration, identifying the data sources which would be required for data mining. Understanding the key features required for modeling from domain point of view.
Audience:

domain experts - for domain knowledge, business rules understanding
IT teams - for data sources identifications, key features of the system
Analytics teams – Data Scientist

Data Acquisition and Data Understanding: This phase involves data collection activities such as pulling data sources from the client databases to a central repository where Data Analytics team develop solutions. Based on the type of problem in hand the data sources might vary from SQL databases to text files to log files to web pages etc.This phase also involves getting familiarity with data - such as data quality, missing values, extracting basic insights and identifying interesting patterns to form Hypothesis.

Data Preparation:This phase of CRISP-DM involves preparing data required to be fed into data mining algorithms. This Phase involves processing or cleaning of raw data. This is one of the crucial steps in data mining. The accuracy of the data mining solution depends on the quality of the data. All the data preparation activities which are required for creating final dataset for feeding into algorithms are done here - Handling missing data using methods such as imputations, converting data into proper formats such as unstructured to structured format, identifying outliers, normalizing the data etc.
Audience:

Data Analytics team

Exploratory Analysis:This is visual analytics phase wherein the data scientist tries to understand the various patterns hiding in the data. The objective of this step is to understand the main characteristics of the data. This analysis is generally done using visualizing tools such as Tableau, R etc.
Performing an Exploratory analysis helps us:

To understand causes of an observed event.
To understand the nature of the data we are dealing with.
Assess assumptions on which our analysis will be based.
To identify the key features in the data needed for the analysis.

Exploratory Analysis involves both Quantitative analysis such as basic statistics such as mean/median/mode, standard deviations etc. and Graphical Analysis such as normal plots, histograms, box plots, scatter plots etc.

Data Modeling:In this phase, various modeling techniques are selected and applied to the data for feature extractions, to model the data, tune the model and to calibrate its parameters to optimal values. Typically this phase involves applying suitable data mining/machine learning algorithms to the dataset. Some problems can be solved using single methods where as some problems involves combination of multiple techniques.
For ex: A recommendation systems of Netflix uses a combination of Boltzman machines, Gradient Boosted Decision trees, logistic regression etc.
Also sometimes different methods are applied separately to select the optimal method to solve the issue at hand.
For ex: Logistic regression, decision tree, Random forests are applied to the dataset to see which model will result in optimal data model.
In this phase of modeling the data, the dataset is divided into two sets, Training Set & Test Set. The modeling the data is done using Training Set and the Test Set is used to evaluate the model.

Data Evaluation: This is the follow-up step to the data Modeling phase. Data Model built in the previous step needs to be thoroughly validated before moving into deployment. The model should address all the business objectives mentioned in the problem statement. The Test Data set created in the previous set is used to test the model build. The objective of this step is to check if the prediction error made on the test set. If the prediction error is less, then our model is good to go. Sometimes the error would be larger indicating the situation of under fitting and Overfitting. Based on the results we might have to go back to previous phases and tune the model.

Deployment:Once the model building and evaluation is completed and we are satisfied with results, the next step is to present the business users with the results. These publishing results should be in user readable or understandable form. Most of the time the results will be published in the form of reports or UI. For example: If the results are needed by the top management for taking key business decisions, visualization reports will be the accurate. If the end user needs to be recommended any new item on e-commerce website, then the results should be displayed on to the web UI.

Most of the time, back and forth between phases is required. For example, during evaluating the data model, if we find that model is suffering from over-fitting we can go back to the model phase and fine tune the Model. As an another example, if in modeling phase if we observe that the a feature column in the dataset with sparse data is very critical in achieving the solution then we will go back to the Business Understanding step and consult the domain experts to know if we can derive more information about the sparse data column and impute the column with relevant values.
To know more information about CRISP-DM, see the wiki page here.

Introduction to Logistic Regression with R

2015-10-07T05:25:00.000+04:00

In my previous blog I have explained about linear regression. In today’s post I will explain about logistic regression.
Consider a scenario where we need to predict a medical condition of a patient (HBP) ,HAVE HIGH BP or NO HIGH BP, based on some observed symptoms – Age, weight, Issmoking, Systolic value, Diastolic value, RACE, etc.. In this scenario we have to build a model which takes the above mentioned symptoms as input values and HBP as response variable. Note that the response variable (HBP) is a value among a fixed set of classes, HAVE HIGH BP or NO HIGH BP.

Logistic regression – a classification problem, not a prediction problem:

In my previous blog I told that we use linear regression for scenarios which involves prediction. But there is a check; the regression analysis cannot be applied in scenarios where the response variable is not continuous. In our case the response variable is not a continuous variable but a value among a fixed set of classes. We call such scenarios as Classification problem rather than prediction problem. In such scenarios where the response variables are more of qualitative nature rather than continuous nature, we have to apply more suitable models namely logistic regression for classification.

Definition:

Assume we have a binary category output variable Y and a vector of p input variables X. Rather than modeling this response Y directly, logistic regression models the conditional probability, Pr(Y = 1|X = x) as a function of x, that Y belongs to a particular category.

Mathematically, logistic regression is expressed as:

Estimating Coefficients – Maximum Likelihood function:
The unknown parameters, β0/ β1, in the function are estimated by maximum likelihood method using available input training data. The Maximum likelihood function expresses the probability of the observed data as a function of the unknown parameters. The maximum likelihood estimators of these parameters are chosen to be those values that maximize this function. Thus, the estimators are those which agree most closely with the observed data.

For now we assume that solving the above equation can be used to estimate the unknown parameters.

In R, we glm() which takes training data as input and gives us the fitted model with estimated parameters as output, which we will see in the later section.

Making Predictions:

Once the coefficients have been estimated, it is a simple matter to compute the probability of response variable for any given input values by putting values of β0/ β1/X in the below equation.

Note: we have predict() in R which takes fitted model, input parameters as input values to predict the response variables.

Use case – Classify if a person may have HBP/not HBP :

Let us take a use case and implement logistic regression in R. Let us classify/predict if a person suffers with High Blood Pressure (HPB) given input predictors AGE, SEX, IsSmoking, Avg Systolic BP, Average diastolic BP, RACE, Body Weight, Height, etc. Access the data set from here. The dataset is NHANES III dataset. In order to implement logistic regression, we need to follow the below steps:

data = read.csv('~\nhanes.csv')

str(data)

data.frame': 15643 obs. of 10 variables:

$ HSAGEIR : int 21 32 48 35 48 44 42 56 82 44 ...

$ HSSEX : int 1 0 0 1 1 1 0 0 0 1 ...

$ BMPWTLBS: num 180 136 150 204 155 ...

$ BMPHTIN : num 70.4 63.9 61.8 69.8 66.2 70.2 62.6 67.6 59.5 71.1 ...

$ TCP : int 268 160 236 225 260 187 216 156 179 162 ...

$ HBP : int 0 0 0 0 0 0 0 0 1 0 ...

$ RACE_2 : num 0 0 0 0 0 1 1 0 0 0 ...

$ RACE_3 : num 0 0 0 0 0 0 0 0 0 0 ...

$ SMOKE_2 : num 0 0 1 0 1 0 0 1 0 0 ...

$ SMOKE_3 : num 0 0 0 0 0 1 1 0 0 1 ...

#removing SEQN,SDPPSU6,SDPSTRA6 from the dataset, from description of the datset,

#we concluded that it is not related to data.

data = data[,-c(1:3)]

#excluding HAR3 for now.

data = data[,-c(1,10)]

# removing entire row if there is any single missing column in the data.

#check for NA's or missing values in the data

#removing all the na rows from data

data = na.omit(data)

# from the description we consider HBP as response variable.

names(data)

'[1] "HSAGEIR" "HSSEX" "DMARACER" "BMPWTLBS" "BMPHTIN" "PEPMNK1R" "PEPMNK5R" "HAR1" "SMOKE" "TCP" "HBP" '

summary(data)

Data Transformations:

'Creating dummy variables/design variables for DMARACER/HSSEX create number of dummy variables based on the levels -1, i.e if if there 3 levels create 3-1 levels of dummy variables x= data

for RACE create 2 dummy variables DMRACE1, DMRACE2 where White(1) - (0,0) black(2) (1,0),other (3) - (1,1)'

for(level in unique(data$DMARACER)){

data[paste("RACE", level, sep = "_")] <- ifelse(data$DMARACER == level, 1, 0)

}

#removing race_1

data = data[,-12]

for(level in unique(data$SMOKE)){

data[paste("SMOKE", level, sep = "_")] <- ifelse(data$SMOKE == level, 1, 0)

}

#removing SMOKE_1

data = data[,-14]

table(data$HSSEX,data$HBP)

     0    1

  0 6576 1702

  1 5817 1548

Fem_HBP = (1702/6576+1702)) = 0.2056052

Mal_HBP = (1548/(1548+5817)) = 0.2101833

Modelling:

#remove HAR1,PEPMNK1R as SMOKE & HBP are created from these variables.

data = data[,-c(3,6,7,8,9,16)]

#create training & testing data

train_x = data

smp_size <- floor(0.75 * nrow(train_x))

train_ind <- sample(seq_len(nrow(train_x)), size = smp_size)

train <- train_x[train_ind, ]

test <- train_x[-train_ind, ]

glm2 = glm(train$HBP~train$HSAGEIR+train$HSSEX+train$BMPWTLBS+train$BMPHTIN+train$TCP+train$RACE_2+train$RACE_3,data=train,family=binomial)

The image shows us the summary of the model. In my next post let us evaluate the logistic regression model and let us consider few other models and choose better model among them.

Exposing R-script as API

2015-04-09T06:20:00.000+04:00

R is getting popular programming language in the area of Data Science. Integrating Rscript with web UI pages is a challenge which many application developers are facing. In this blog post I will explain how we can expose R script as an API, using rApache and Apache webserver.

rApache is a project supporting web application development using the R statistical language and environmentand the Apache web server.

Exposing Rscipt as API typically involves 3 steps:

Pre-requisites
Installing rApache
Configuring rApache

Step #1 - Pre_requisites :

Linux environment - ubuntu
latest version of R - R- 3.1.0

Step#2 - Install apache webserver as below:

apt-get install r-base-dev apache2-mpm-prefork apache2-prefork-dev
wget http://www.rapache.net/rapache-1.2.3.tar.gz
tarxzvf rapache-1.2.3.tar.gz
cd rapache-1.2.3
./configure
make
make install

Step#3: Configuring rApache.

Once rApache is installed, start the rApache server as below:

service apache2 start

Since we are installing for the first time let us test if the setup is installed properly. Do the below steps.

sudo vim /etc/apache2/apache2.conf #Added the following:

Once you have done the above step, Now access the below pages:

IPAddress/html/index.html

IPAddress/RApacheInfo.html

Before getting into expose R-script as API let us understand the rApache configuration.
Once the rApache is installed the folder structure of rApache would be as below:

In apache2.conf file, we need to configure all the sites/r-scripts that we need to expose as api.. Below we have set a directory at path “/var/www/RProject“ in directory tags in apache2.conf file.

Now place the below R-script (test.r) generating a random normal distribution in the RProject folder specified above,in the code belowGET$p is input parameter we pass from URL.

The api is ready for testing.We can access the api using IP.XXX.XXX.XXX/RProject/test.r?q=10 And the results would be as below:

Detailed documentation of Installation is found here

Regression Analysis using R

2014-10-05T00:35:00.000+04:00

What is a Prediction Problem?

A business problem which involves predicting future events by extracting patterns in the historical data. Prediction problems are solved using Statistical techniques, mathematical models or machine learning techniques.

For example: Forecasting stock price for the next week, predicting which football team wins the world cup, etc.

What is Regression analysis, where is it applicable?

While dealing with any prediction problem, the easiest, most widely used yet powerful technique is the Linear Regression. Regression analysis is used for modeling the relationship between a response variable and one or more input variables.

In simpler terms,Regression Analysis helps us to find answers to:

Prediction of Future observations
find association, relationship between variables.
Identify which variables contribute more towards predicting the future outcomes.Types of regression problems:

Simple Linear Regression:

If model deals with one input, called as independent or predictor variable and one output variable, called as dependent or response variable then it is called Simple Linear Regression. In this type of Linear regression, it assumes that there exists a linear relation between predictor and response variable of the form.

Y ≈ β0 + β1X + e.

In the above equation, β0,β1 are the unknown constants that represent intercept and slop of a straight line which we learned in our high schools. These known constants are known as the model coefficients or parameters. From the above equation, X is the known input variable and if we can estimate β0,β1 by some method then Y can be predicted. In order to predict future outcomes, by using the training data we need to estimate the unknown model parameters (ˆ β0,ˆ β1) using the equation.

ˆy = ˆ β0 + ˆ β1x + ˆe, where ˆ y,ˆ β0,ˆ β1 are the estimates.

Multiple Linear Regression:

If the problem contains more than one input variables and one response variable, then it is called Multiple Linear regression.

How do we apply Regression analysis using R?

Let us apply regression analysis on power plant dataset available from here. The dataset contains 9568 data points collected from a Combined Cycle Power Plant over 6 years (2006-2011), when the power plant was set to work with full load. Features consist of hourly average ambient variables Temperature (T), Ambient Pressure (AP), Relative Humidity (RH) and Exhaust Vacuum (V) to predict the net hourly electrical energy output (EP) of the plant.

Read the data into R environment:

require(xlsx)

sample1 = read.xlsx("C:\\Suresh\\blogs\\datasets\\CCPP\\Folds5x2_pp.xlsx",sheetIndex=1)

Understand and observing the data: View(sample1)

Check for missing values, range of variables, density plots for each of the varaible:

sum(is.na(sample1$AT))

[1] 0

range(sample1$AT) #1.81,37.11

mean(sample1$AT) #m: 19.65

plot(density(sample1$AT))

Plot(sample1)

Density plot for Temperature

Scatter plots shows us that temperature (AT) and vaccum(V) are inversely related to power while pressure(AP) and RH are not related.

Check for correlation among the variables. This step is very important to understand the relation of dependant variable with the independent variables and correlations among the variables. In general, there shouldn’t be any correlation among the independent variables.

cor(sample1)

'AT          V          AP          RH         PE

AT  1.0000000  0.8441067 -0.50754934 -0.54253465 -0.9481285

V   0.8441067  1.0000000 -0.41350216 -0.31218728 -0.8697803

AP -0.5075493 -0.4135022  1.00000000  0.09957432  0.5184290

RH -0.5425347 -0.3121873  0.09957432  1.00000000  0.3897941'

'inferences--> AT has -ve relation with PE

V is highly related to PE

other two are relatively related'


Divide the data into training and test set and train the model with linear regression using lm()  method available in R and thendo predictions on new test data using predict() method.

rand=sample1[sample(nrow(sample1)),]

tr = rand[1:6697,]

ts = rand[6698:9568,]

model2 = lm(PE~AT+V+AP+RH,data=tr)

summary(model2)

Call:

lm(formula = PE ~ AT + V + AP + RH, data = tr)

Residuals:

Min 1Q Median 3Q Max

-43.533 -3.170 -0.068 3.229 17.451

Coefficients:

Estimate Std. Error t value Pr(>|t|)

(Intercept) 457.729155 11.794172 38.810 < 2e-16 ***

AT -1.987307 0.018208 -109.147 < 2e-16 ***

V -0.231996 0.008692 -26.689 < 2e-16 ***

AP 0.059235 0.011442 5.177 2.32e-07 ***

RH -0.159916 0.005015 -31.886 < 2e-16 ***

---

Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Residual standard error: 4.585 on 6692 degrees of freedom

Multiple R-squared: 0.9281, Adjusted R-squared: 0.9281

F-statistic: 2.161e+04 on 4 and 6692 DF, p-value: < 2.2e-16

New predictions are made using predict method.

pred = predict(model,ts[,1:4])

The above image shows the results of actual vs predicted which are quite accurate. In the summary results of the model, below are the key takeways:

Model is accurate as R²is near to 1 (0.912).
Model states all the variables are significant, the *** indicate the significance.
P-statistics is less than 0.05, F-statistics is significantly high.
Residuals vs fitted plots and Q-Q normal plots are also good with mean variance of the errors around 0.

In the next blog we learn about model validations and extensions in linear regression.

Assessing Model Accuracy - Part 2

2014-07-31T12:35:00.000+04:00

In my last post, I have explained about MSE, today I will explain the variance & bias trade-off, Precision recall trade-off while assessing the model accuracy.

What is Variance and bias of a statistical learning Method?
Variance refers to the amount by which the estimated output (f) would change if we estimated it (f) using a different training dataset. Since the training data is used to fit the statistical learning method, different training sets will result in different outputs (f).

Ideally, the estimate should not vary much between training sets.
Bias refers to the error that is introduced by approximating a complicated problem by a simpler model.
For example: Consider the distribution of dataset (black fit curve). If a simple linear regression (orange fit curve) is fitted for a dataset which actually needs much flexible model (blue fit curve), the simple linear regression model induces Bias in the model.

Fig: linear regression provides a very poor fit to the data

Explanation:
The test MSE calculated here, can be decomposed into three properties, Variance, Bias, error.

E(y0 − ˆ f(x0))^2 = Var( ˆ f(x0)) + [Bias( ˆ f(x0))]^2 +Var(ɛ).

Where E(y0 − ˆ f(x0))^2 defines the expected test MSE, first part of the equation is the variance, second part is the bias and the third part is the variance of error.
In order to develop best model for any analysis, we need to select a statistical method which achieves low Variance and low bias. This is called Variance-bias trade-off.

Fig: Variance-bias trade-off - Below explanation explains the details.

As a general rule, as the complexity of the model increases the variance will increase & Bias will decrease. The relative rate of change of these two quantities determines whether the test MSE increases or decreases. Initially, as the flexibility increases the bias decreases faster than the variance increases. As a result, test MSE decreases. However, at some point increasing flexibility has little effect on the bias and after this point the variance tends to increase significantly. This point can be treated as optimal point for model selection. As an end note, while assessing the model accuracy we need to take variance-bias trade-off into consideration.

What is Precision-Recall of a statistical learning Method?
While dealing with a classification problem, we validate the relevance of the model using Precision Recall methods.
Consider the below confusion matrix, one of the best ways to represent the results of a classifier:

		ACTUAL
		POSITIVE	NEGATIVE
PREDICTED	POSITIVE	TRUE POSITIVE	FALSE POSITIVE
PREDICTED	NEGATIVE	FALSE NEGATIVE	TRUE NEGATIVE

Let’s understand the confusion matrix:
TRUE POSITVE: The actual is +ve and model predicted +ve.
FALSE POSITIVE: The actual is –ve but the model predicted positive, FALSE ALARM
FALSE NEGATIVE: The actual is +ve but the model predicted negative - A MISS
TRUE NEGATIVE: The actual is –ve & model predicted –ve.
Precision: % of the predicted values that are correct.

Precision = TP/(TP+FP)

Recall: % of the correct items that are relevant.

Recall = TP/(TP+FN)

Both Precision & Recall are inversely related, if precision increases, the recall falls, vice versa. Trade-off between Precision and Recall varies from problem to problem. For example, in Legal document classification problem, model needs to have high Recall, as the model needs to extract/classify more relevant.

Assessing Model Accuracy - Part1

2014-06-21T22:42:00.000+04:00

Recently, I have started reading a book "Introduction to statistical Learning", which had good introduction for model accuracy assessing. This post contains excerpts of the chapter:

Often we take different statistical approaches to build a solution for a data analytical problem. Why is it necessary to introduce so many different approaches, rather than a single best method? The answer is: in Statistics no single method dominates all other methods over all possible datasets. One statistical method may work well with a specific dataset and some other method may work better on a similar but different dataset. So it is important to decide for a particular dataset which method produces best results.

In order to evaluate the performance of a statistical learning method, we need to measure how well its predictions actually match the observed data. Today we will look into differnt measures which help us in assessing the model accuracy:

Mean Squared Error
Variance/bias

Mean Squared Error:

In regression Analysis, the most commonly used measure is Mean Squared Error, given by the equation:

where f(xi) is the prediction that f gives for the ith observation.

The MSE will be small if the predicted responses are very close to the true responses, and will be large if for some of the observations, the predicted and true responses differ substantially. The MSE calculated above is called training MSE. But here we need to make a note that we would be more interested in the accuracy of test MSE, i.e. the MSE of previously unseen test data on the model rather than training MSE. The equation for calculating the test MSE is

Test MSE =Avg(f(x0)-y0)^2

Where test MSE is calculated by squared average of the difference between the actual (y0) & estimated f(x0) on.

Consider the below situation where 3 different regression model fits are applied on a randomly generated data set and is shown in below fig:

Note: The above images are taken from An Introduction to Statistical Learning - Authors: Gareth James, Daniela Witten, Trevor Hastie and Robert Tibshirani

In the above figure, black curve is actual distribution of data, orange, blue, green are three different fits to the original data distribution. For the right portion of the image we can say that the model fit representing the blue curve is more near to the original & it is the ideal model for prediction.

Now consider left part of image, the red curve is the test MSE for three model fits & the grey curve is the training MSE for the same three model fits. We can observe that the grey curve, the training MSE will keep on declining telling us that with more complex models than required, the MSE will be high. On the other hand, the test MSE is a U shaped curve which decreases initially and then raises. The ideal model is the one after which the test MSE increases. In our example it is at the Blue point where the curve starts raising. The model we built should not result in Overfitting, a condition in which the method yields small training MSE but a large test MSE. It means that the modelled fit/method is trying to so hard to find patterns in the data and fitting up even though the occurrences were by chance in reality.

In my next post will explain about Variance-Bias trade-off while choosing a best fit model.

Basic recommendation engine using R

2014-05-25T11:03:00.000+04:00

In our day to day life, we come across a large number of Recommendation engines like Facebook Recommendation Engine for Friends’ suggestions, and suggestions of similar Like Pages, Youtube recommendation engine suggesting videos similar to our previous searches/preferences. In today’s blog post I will explain how to build a basic recommender System.

Types of Collaborative Filtering:

User based Collaborative Filtering
Item based Collaborative filtering

In this post will explain about User based Collaborative Filtering. This algorithm usually works by searching a large group of people and finding a smaller set with tastes similar to yours. It looks at other things they like and combines them to create a ranked list of suggestions.

Implementing User Based Collaborative Filtering:
This involves two steps:

Calculating Similarity Function
Recommend items to users based on user Similarity Score

Consider the below data sample of Movie critics and their movie rankings, the objective is to recommend the unrated movies based on similar users:

Step1- Calculate Similarity Score for CHAN:

Creating Similarity score for people helps us to identify similar people. We use Cosine based Similarity function to calculate the similarity between the users. Know more about cosine similarity here. In R we have a cosine function readily available:

user_sim = cosine(as.matrix(t(x)))

Step2- recommending Movies for CHAN:

For recommending movies for Chan using the above similarity matrix, we need to first fill the N/A where he has not rated. As first step, separate the non-rated movies by Chan and a weighted matrix is created by multiplying user similarity score (user_sim[,7]) with ratings given by other users.

Next step is to sum up all the columns of the weight matrix, then divide by the sum of all the similarities for critics that reviewed that movie. The result calculation gives what the user might rate this movie, the results as below:

The above explanation is written in the below R function:
rec_itm_for_user = function(userNo)
{ #calcualte column wise sum
col_sums= list()
rat_user = critics[userNo,2:7]
x=1
tot = list()
z=1
for(i in 1:ncol(rat_user)){
if(is.na(rat_user[1,i]))
{
col_sums[x] = sum(weight_mat[,i],na.rm=TRUE)
x=x+1
temp = as.data.frame(weight_mat[,i])
sum_temp=0
for(j in 1:nrow(temp))
{ if(!is.na(temp[j,1]))
{
sum_temp = sum_temp+user_sim[j,7]
}
}
tot[z] = sum_temp z=z+1
}
}
z=NULL
z=1
for(i in 1:ncol(rat_user)){
if(is.na(rat_user[1,i]))
{
rat_user[1,i] = col_sums[[z]]/tot[[z]] z=z+1
}
}
return(rat_user)
}
Calling the above function gives the below results:

rec_itm_for_user(7)

Titanic Batman Inception Superman.Returns spiderMan Matrix

2.811 4.5 2.355783 4 1 3.481427
Recommending movies for Chan will be in the order: Matrix (3.48), Titanic(2.81), Inception(2.35).

complete sourceCode is available on github

Time Series Analysis using R - forecast package

2014-04-17T11:26:00.003+04:00

In today’s blog post, we shall look into time series analysis using R package – forecast. Objective of the post will be explaining the different methods available in forecast package which can be applied while dealing with time series analysis/forecasting.

What is Time Series?

A time series is a collection of observations of well-defined data items obtained through repeated measurements over time. For example, measuring the value of retail sales each month of the year would comprise a time series.

Objective:

Identify patterns in the data – stationarity/non-stationarity.
Prediction from previous patterns.

Time series Analysis in R:

My data set contains data of Sales of CARS from Jan-2008 to Dec 2013.

Problem Statement: Forecast sales for 2013

MyData[1,1:14]

PART	Jan08	FEB08	MAR08	....	....	NOV12	DEC12
MERC	100	127	56	....	....	776	557

Table: shows the first row data from Jan 2008 to Dec 2012

Results:

The forecasts of the timeseries data will be:

Assuming that the data sources for the analysis are finalized and cleansing of the data is done, for further details,

Step1: Understand the data:

As a first step, Understand the data visually, for this purpose, the data is converted to time series object using ts(), and plotted visually using plot() functions available in R.

ts = ts(t(data[,7:66]))
plot(ts[1,],type=’o’,col=’blue’)

Image above shows the monthly sales of an automobile

Forecast package & methods:

Forecast package is written by Rob J Hyndman and is available from CRAN here. The package contains Methods and tools for displaying and analyzing univariate time series forecasts including exponential smoothing via state space models and automatic ARIMA modelling.

Before going into more accurate Forecasting functions for Time series, let us do some basic forecasts using Meanf(), naïve(), random walk with drift – rwf() methods. Though these may not give us proper results but we can use the results as bench marks.

All these forecasting models returns objects which contain original series, point forecasts, forecasting methods used residuals. Below functions shows three methods & their plots.
Library(forecast)
mf = meanf(ts[,1],h=12,level=c(90,95),fan=FALSE,lambda=NULL)
plot(mf)

mn = naive(ts[,1],h=12,level=c(90,95),fan=FALSE,lambda=NULL)
plot(mn)

md = rwf(ts[,1],h=12,drift=T,level=c(90,95),fan=FALSE,lambda=NULL)
plot(md)

Measuring accuracy:

Once the model has been generated the accuracy of the model can tested using accuracy(). The Accuracy function returns MASE value which can be used to measure the accuracy of the model. The best model is chosen from the below results which gives have relatively lesser values of ME,RMSE,MAE,MPE,MAPE,MASE.
> accuracy(md)
ME RMSE MAE MPE MAPE MASE
Training set 1.806244e-16 2.445734 1.889687 -41.68388 79.67588 1.197689
accuracy(mf)
ME RMSE MAE MPE MAPE MASE
Training set 1.55489e-16 1.903214 1.577778 -45.03219 72.00485 1
> accuracy(mn)
ME RMSE MAE MPE MAPE MASE
Training set 0.1355932 2.44949 1.864407 -36.45951 76.98682 1.181666

Step2: Time Series Analysis Approach:

A typical time-series analysis involves below steps:

Check for identifying under lying patterns - Stationary & non-stationary, seasonality, trend.
After the patterns have been identified, if needed apply Transformations to the data – based on Seasonality/trends appeared in the data.
Apply forecast() the future values using Proper ARIMA model obtained by auto.arima() methods.

Identify Stationarity/Non-Stationarity:

A stationary time series is one whose properties do not depend on the time at which the series is observed. Time series with trends, or with seasonality, are not stationary.
The stationarity /non-stationarity of the data can be known by applying Unit Root Tests - augmented Dickey–Fuller test (ADF), Kwiatkowski-Phillips-Schmidt-Shin (KPSS) test.

ADF: The null-hypothesis for an ADF test is that the data are non-stationary. So large p-values are indicative of non-stationarity, and small p-values suggest stationarity. Using the usual 5% threshold, differencing is required if the p-value is greater than 0.05.

library(tseries)

adf = adf.test(ts[,1])

adf

Augmented Dickey-Fuller Test

data: ts[, 1]

Dickey-Fuller = -4.8228, Lag order = 3, p-value = 0.01

alternative hypothesis: stationary

The above figure suggests us that the data is of stationary and we can go ahead with ARIMA models.

KPSS: Another popular unit root test is the Kwiatkowski-Phillips-Schmidt-Shin (KPSS) test. This reverses the hypotheses, so the null-hypothesis is that the data are stationary. In this case, small p-values (e.g., less than 0.05) suggest that differencing is required.

kpss = kpss.test(ts[,1])

Warning message:

In kpss.test(ts[, 1]) : p-value greater than printed p-value

kpss

KPSS Test for Level Stationarity

data: ts[, 1]

KPSS Level = 0.1399, Truncation lag parameter = 1, p-value = 0.1

Differencing:
Based on the unit test results we identify whether the data is stationary or not. If the data is stationary then we choose optimal ARIMA models and forecasts the future intervals. If the data is non- stationary, then we use Differencing - computing the differences between consecutive observations. Use ndiffs(),diff() functions to find the number of times differencing needed for the data & to difference the data respectively.

ndiffs(ts[,1])

[1] 1

diff_data = diff(ts[,1])

Time Series:

Start = 2

End = 60

Frequency = 1

[1] 1 5 -3 -1 -1 0 3 1 0 -4 4 -5 0 0 1 1 0 1 0 0 2 -5 3 -2 2 1 -3 0 3 0 2 -1 -5 3 -1

[36] -1 2 -1 -1 5 -2 0 2 -2 -4 0 3 1 -1 0 0 0 -2 2 -3 4 -3 2 5

Now retest for stationarity by applying acf()/kpss() functions if the plots shows us the Stationarity then Go ahead by applying ARIMA Models.
Identify Seasonality/Trend:
The seasonality in the data can be obtained by the stl()when plotted

Stl = Stl(ts[,1],s.window=”periodic”)

Series is not period or has less than two periods

Since my data doesn’t contain any seasonal behavior I will not touch the Seasonality part.

ARIMA Models:
For forecasting stationary time series data we need to choose an optimal ARIMA model (p,d,q). For this we can use auto.arima() function which can choose optimal (p,d,q) value and return us. Know more about ARIMA from here.

auto.arima(ts[,2])

Series: ts[, 2]

ARIMA(3,1,1) with drift

Coefficients:

ar1 ar2 ar3 ma1 drift

-0.2621 -0.1223 -0.2324 -0.7825 0.2806

s.e. 0.2264 0.2234 0.1798 0.2333 0.1316

sigma^2 estimated as 41.64: log likelihood=-190.85

AIC=393.7 AICc=395.31 BIC=406.16

Forecast time series:

Now we use forecast() method to forecast the future events.

forecast(auto.arima(dif_data))

   Point Forecast     Lo 80      Hi 80     Lo 95    Hi 95

61   -3.076531531 -5.889584 -0.2634795 -7.378723 1.225660

62    0.231773625 -2.924279  3.3878266 -4.594993 5.058540

63    0.702386360 -2.453745  3.8585175 -4.124500 5.529272

64   -0.419069906 -3.599551  2.7614107 -5.283195 4.445055

65    0.025888991 -3.160496  3.2122736 -4.847266 4.899044

66    0.098565814 -3.087825  3.2849562 -4.774598 4.971729

67   -0.057038778 -3.243900  3.1298229 -4.930923 4.816846

68    0.002733053 -3.184237  3.1897028 -4.871317 4.876783

69    0.013817766 -3.173152  3.2007878 -4.860232 4.887868

70   -0.007757195 -3.194736  3.1792219 -4.881821 4.866307

plot(forecast(auto.arima(dif_data)))

The below flow chart will give us a summary of the time series ARIMA models approach:

The above flow diagram explains the steps to be followed for a time series forecasting

Build Web applications using Shiny R

2014-03-20T00:05:00.001+04:00

Ever since I’ve started working on R , I always wondered how I can present
the results of my statistical models as web applications. After doing some
research over the internet I’ve come across – ShinyR – a new package
from RStudio which can be used to develop interactive web applications with R.
Before going into how to build web apps using R, let me give you some overview
about ShinyR.

Features:

No JavaScript/HTML knowledge required.
Can build interactive web applications which can load the data dynamically.
Pre-Built output widgets for displaying plots, tables interactively are available.
Works in any R environment.
Shiny Apps can be hosted locally/on self-hosted Shiny Server/Rstudio-hosted Shiny Server – in today’s post we will see hosting locally.

Building a shiny App

Shiny application have two components: a user-interface definition and server script,
and any additional data, scripts, or other resources required to support the
application.
UI script (ui.R): In this script we define the UI layout of the webpage. We can make
use of inbuilt UI controls/widgets for displaying output and for creating UI layout..
Server Script (server.R): In this script, the actual business logic/ code for statistical
models, plotting the output is defined. It accepts inputs from UI layer and compute
outputs send back the response to the UI layer. The basic task of a Shiny server
script is to define the relationship between inputs and outputs.

Understanding UI.R script:

library(shiny) #load shiny package
ds = read.csv("inputdata.csv") #load data
x = names(ds) # Define UI for application that plots distributions
shinyUI(pageWithSidebar(
# Application title
headerPanel("Demo Shiny Application"),
#sidebar panel to the left side of the page for input controls, slider & dropdown
sidebarPanel(
sliderInput('Year', 'Year', min=2010, max=2012,
value=min(2010,2012), step=1, round=0),
selectInput('select', 'Select Criteria', c(x[3],x[9],x[10],x[11],x[12],x[26]))
),
#main panel for displaying outputs, Histograms, summary
mainPanel(
plotOutput('plot'),
verbatimTextOutput("summary")
) ))

Notice in particular, in the above code sidebarPanel and mainPanel functions
are called with two arguments - corresponding to the two inputs and
two outputs displayed.

In the above image we can see the layout which contains a slider with year range,
where based on the year selection the dataset will be taken into consideration.
Dropdownlist containing varaibles, placeholder to display the graphs, table to
display the summaries of the models.

Understanding Server.R

In the above script, we can observe that all the logic for the exploratory analysis is

written in server.R script. It contains a main function called ShinyServer()

wherein we can write reactive functions,reactive plots, outputs. The outputs

are send back to the UI layer.

In the above code, I have written logic which contains data fetching,

modification, setting the dataset based on the year selected in the

slider (reactive function), displaying histogram based on the selection

on slider and dropdownlist,writing summary to a table.

Deploying the application:

Once we are done with the application, next step is to deploy the application.

Using Shiny, we have two different options to move the application to Production.

** Create a local- self-hosted Shiny Server on Linux platform, here.
** Host on the R-Studio's Shiny server, explained here

Deploying locally:

As mentioned in the beginning of the post, in order to deploy locally, we make

use of GIST. The steps are quite simple:

** Upload the Server.R & UI.R files to the Gist.Github.com website.
** Uploading the files gives you an ID - in my case - I got "8942533"
** After finishing upload, open R and run the below code:

library('shiny')
shiny::runGist('8942533')

The model will be shown as web application will be displayed

the browser with inputs for selection & outputs to display.

Exploratory data analysis techniques

2014-03-03T18:08:00.000+04:00

In my previous blog post I have explained the steps needed to solve a data analysis problem. Going further, I will be discussing in-detail each and every step of Data Analysis. In this post, we shall discuss about exploratory Analysis.

What is Exploratory Analysis?

“Understanding data visually”

Exploratory Analysis means analyzing the datasets to summarize their main characteristics, often visually. This is the first step of any data analysis.

Objectives:

Know the data types of the dataset – whether continuous/discreet/categorical
Understand how the data is distributed
Extract important input variables for the analysis
Identify outliers
To identity patterns, if exists

Exploratory Analysis Techniques:

Box-Plot
Histogram
Trend analysis
Scatter Plots

Let us understand the exploratory analysis by considering a data analysis problem.

Problem statement:

to analyze the incidents/events occurred over past 3 years and try to predict the event occurring in the future.

Solution:

After understanding the problem statement and gaining the sufficient domain knowledge, Identify the data sources & download the data into the programming environment.

The next step is to perform an Exploratory analysis as explained here. in today's post we shall look how exploratory analysis can be done.

Types of Exploratory analysis:

Type1: Understanding the data – variable names, dimensions of the dataset, data types of each and every variable.

data = read.csv("datasource.csv") #load data

dim(data) #know the dimensions of the data

[1] 839 50

Colnames(data) #know the column names

[1] "Incidents" "Year of Occurance" "Location.of.Occurrence" "Date.of.Occurence" [5]"Time.of.Occurrence" “Operational Phase”

Str(data) # know the data types of each of the variable – continuous/descrete/categorical

$ Incidents: int 41505 41537 41539 41565 41589 41596 41598

$ Vehicle.Type : Factor w/ 7 levels "","Volvo(all series)",..: 6 2 2 2 6 6

$ Location.of.Occurrence: Factor w/ 101 levels "","Abidjan","Accra",..: 53 35 35 35 96

$ Date.of.Occurence: Factor w/ 520 levels "1/1/2010","1/1/2012" ..: 1 32 37

Sum(!is.na(data$Date.of.Occurance) # counting the number of missing values in the column

Type2:Creating new varaibles/data type conversion suitable for the analysis – like factor variables into numerical,dates into year/month/day,time into hour of the day, etc. according to our convineince.

#Extracting year/month from Date of occurrence and creating new variables

xn = as.POSIXct(data$"Date.of.Occurence",format="%m/%d/%Y")

data["year"] = as.numeric(format(xn,"%Y"))

data["month"] = as.numeric(format(xn,"%m"))

str(data$"year")

num [1:839] 2010 2010 2010 2010 2010 2010

#extracting hour of the day and creating new variable TimeOfOccurance

(TOC) data["TOC"] = sub(":.*", "", data$"Time.of.Occurrence")

str(data$"TOC")

chr [1:839] "18" "6" "21" "13" "16" "13" "11" "11" "15" "1" "13" "6"

Type3: Observe the summary of each and every variable to understand the variables. summary(data$"Vehicle.Type")

Volvo (all series) ASHOK (all series) FIAT (all series) Maruthi (all series)

210 49 71 39

Type4: Decide which variables are good for analysis by using trends, boxplots, histograms etc.

boxplot(formula=as.numeric(data$"operational.Phase")~data$"year",col="blue")

Box plot distribution of incidents occurring over the years.

hist(data[which(data$"year" == 2011),]$"month",breaks = "Sturges",col=c('blue','red','green'),labels=T)

The above histogram depicts the month wise distribution of incidents occurred in 2011 Trend analysis

In the above graph, we can bring out the below inferences:

Sharp fall in the data in 2012 might be not capturing of the incidents

An average of 30 incidents occurring monthly

In the month of Feb there is sharp fall in the incidents

In the above trend image with graph in red color is plotted against Number of people in the deck and number of Incidents.

This clearly says that there is no relation between the Incident occurring and number of people in the deck.

Hope the above post gives you a very good understanding of how exploratory analysis can be done. In my next post we shall learn how to do forecasting using Linear regression.

Data Analysis Steps

2014-02-03T14:06:00.001+04:00

After going through the overview of tools & technologies needed to become a Data scientist in my previous blog post, in this post, we shall understand how to tackle a data analysis problem.
Any data analysis project starts with identifying a business problem where historical data exists. A business problem can be anything which can include prediction problems, analyzing customer behavior, identifying new patterns from past events, building recommendation engines etc.

The steps for solving a data analysis problem can be shown as below:

Identify Business Problem:

“Define Problem statement”

This is the first step of analysis. Business identifies a problem and a problem statement with desired outcome is defined. In this stage, a Data Scientist should understand the problem statement, the domain knowledge of the problem. After thorough understanding of the problem statement, a Hypothesis will be proposed.

Data Acquisition:

“Identify data sources”

As a second step, all the data sources related to the problem statement will be identified and pulled into a central repository. The data sources can vary from SQL databases to text files to csv files to online data. If the data size is large we may use Hadoop to pull, store & pre-process the data.

Process/Clean Data:

“The accuracy of the results of analysis depends on the quality of data”

Data Clean step is considered to be one of the very important phases in Data analysis. The accuracy of the analysis depends on the quality of data.
Few approaches:

Formatting the data as per the data analytical tools we use.
Missing data handling
Data Transformations like normalizing the data Identifying outliers & handling etc.

Exploratory Analysis:

“Embrace the data visually before diving further”

The objective of this step is to understand the main characteristics of the data. This analysis is generally done using visualizing tools. Performing an Exploratory analysis helps us:

to understand causes of an observed event
to understand the nature of the data we are dealing with
assess assumptions on which our analysis will be based
to identify the key features in the data needed for the analysis

Graphical Techniques:Scatter plots, box plots, histograms
Quantitative techniques: Mean, median, Mode, Standard deviation

Model Generation & Validation:

“Select-Train-Evaluate”

This step involves extracting features from the data and feeding them into the machine learning algorithms to build a model. Model is the solution proposed for the problem statement. This step involves: Model selection, model training and model evaluation.
Model selection: Based on the type of business problem we are dealing, a model will be built. For example,if the objective of the analysis is to predict a future event, we need to build a Regression model for prediction.
Model Training: After selecting the Model for the analysis, the entire dataset is divided into 2 parts – Training data & Test Data. 3/4th of the entire data will be fed as input to the Model Algorithms.
Model Evaluation: Once the model is built. The next step is to test the model & validate it. The data used for testing the model is the remaining 1/3rd of the dataset in the previous step.

Visualize Results:

"Show the results visually"

This is the final step of Data analysis where the results of the model & problem solved will be presented generally in visual plots/graphs.
Few visualizing tools: d3.js, ggplot2, tableau.

Please go through the tools/technologies , skill set required to learn Data Analysis here

Data Analysis Tools

2014-01-07T16:25:00.000+04:00

As mentioned in my previous post , in this post I will be listing out the tools, blogs and forums, online courses that I have gathered over the past one year, which I felt necessary in my journey, which will be helpful to my fellow data science aspirants.

Skillset Required:

Knowledge in Statistics – Exploratory analysis, doing initial analysis of the data & understanding the data to decide what techniques needs to be applied, which I feel is a must know subject.
Mathematics – basics of calculus, algebra etc. for mathematical formulation of the problem statement.
Understanding Machine learning algorithms for predictive modeling, recommendation engines, classification models, cluster analysis, social network analysis.
Data mining skills like data cleaning/Data Munging skills, apply Machine learning techniques on the data.
Visualization skills to display the results, to understand the results during building data modeling.

Tools required:
Programming Languages: Proficiency in any two of the below mentioned languages would be advisable:

R
Python
Java – comes in handy when we work on Hadoop
C,C++

Tools required: Since I’m using Open source tools, I will be confined to them:

R-Studio
NLTK toolkit
Rapid Miner
Weka

Important Point:
Most of the machine learning algorithms has been already implemented as packages in the above languages/tools . We need to just download and make use of them.
Big data Tools:

Hadoop setup from Cloudera/Hortonworks
Mongodb- NoSQL DB
HBASE, PIG, HIVE.

Visualization tools:
Though I have not explored much in this area, but till day I’m happy with R packages for visualizations.

Data exploration in R/Python

Few Books I have referred:

The Elements of Statistical Learning - 2nd Edition
Simon Sheather, A Modern Approach to Regression
Data Mining 3rd Edition by Ian H. Witten, Eibe Frank, Mark A. Hall

Online Courses:
Though a lot of courses are available online, I have stick to very few sites as below,
For Data Analysis, Stats, Maths:

For Big Data:

Blogs and forums:
Online forums is one place where I used to get a lot of information, in Linked Groups I could get answer to all my trivial questions. You post any query and you will get elaborate answer from research scholars to industry experts, I really love this place. I will list down few Linkedin groups I follow,

Blogs I follow:

Will add more when I come across the new tools. Guys hope this will serve you as a starting point for the Journey. All the best, Happy New Year. Please do add any new tools and technologies to the above list.
In my next post, shall post about how to tackle a data analysis problem

Data Scientist. The Path I Chose

2013-12-31T09:12:00.000+04:00

As we all are marching into the New Year, I would like to post about my plans to become a Data Scientist, my 2014 resolution at Professional front. The term Data Science was first introduced to me a year ago same time. Since then I have started researching and gathering necessary information and decided to become Data Scientist. After one year I just wanted to look back myself to understand where I stand now & still what needs to be done.
Power of Data – possible Use Cases:

Automobile companies are installing special devices which continuously feed data about the nature of driving, the roads being taken, places being visited etc. to the Insurance companies who by analyzing this information can customize the insurance plans for its customers.
Another possible use case, making use of the data we store on social media, we can develop products which can understand us and sometimes guide us. Like, based on people’s posts on social media, with location, time, road you are travelling & LIVE Traffic feeds, we can develop an app which could suggest us which road to take to our destination taking.
Who is Data Scientist?
Data Scientist termed as the Sexiest Job of the 21st Century and is the most sought after & well paid job all over the Industry. The role of the Data Scientist is to Make Sense out of the tons of Data being generated every day, build products which can redefine more statistically/ scientifically the way the current businesses is going. The best part of this Job Is, a Data Scientist can fit into any industry where ever data exists.
What it needs to become a Data Scientist?

"Web is my University, Time is the only Investment"

Going back to where I have started my journey, after my initial research with my folks and over internet, I started taking up Machine Learning course from Coursera.org, an online offering from Stanford University, definitely a recommended Starting Point for anyone who wants to be a Data Scientist.
After completing the course, I have started playing with Data. Predictive models, being my first Hands-on. At this point I got an advice from my Colleague,

“It is easy (because free and online) in order to grab some interesting key messages and get a rather complete overview within a 8-week period timeframe. Be sure you have a sufficient mathematical background to follow (maybe Calculas/mathematics before?) and more importantly, be certain you can practice afterwards!”

My First analytics assignment on Predictive modelling gave me a lot of learnings, the need to brush up my Mathematics basics, Stats fundamentals, data mining techniques, Visualization Techniques.
I was lucky enough to find many of the courses listed in my TODO list
@ Coursera.org, UDACITY:

Machine Learning – for Machine Learning Algorithms
Neural Networks for Machine Learning
Data Analysis – Introduction to Programming Language R
Language for Data Analysis
Making Sense of Data - basics of Statistics
Introduction to Data Science – intro to Data Science
Social Network Analysis

The image, below which I have got over internet, is a simplified road map for a Data Scientist, which has become my Target over the next few months combined with Hands-on.

Road map to data science

After completing the required courses mentioned above, I got an opportunity to play around with Data using Text Mining, Classification techniques, Predictive modelling techniques both from within my Organization & outside (KAGGLE.COM).
It would be incomplete, if I don’t mention KAGGLE, the website where I found the much needed hands-on experience. KAGGLE is an online Open competition forum, where you can find Data Science Use cases posted by big Companies. Advantage of working on this site is not only you can get hands-on experience but also you increase your Network with similar competent techies.
Apart from KAGGLE, I have made use of LINKEDIN data science/data Analysis groups where I could get suggestions and answers to almost every question of mine by the Experts of this field.
Data Analytics Should Meet Big Data:

“Learn a bit of Java, Learn a bit of Linux – shell scripting, Learn Hadoop” – an advice from my friend.

After completing few POCs and began working with real time projects I have soon realized the need to handle huge amounts of data and need to learn BIG DATA. In my personal opinion, a Data Scientist should have a very strong knowledge in Big Data-Hadoop.
For this, I planned to choose the road path of learning Big Data Hadoop concepts, Linux Shell scripting, MongoDB for data storage in the next 3 months.
That is me folks, after one year; long way to go through before becoming a Data Scientist. But still would love to hear someone calling me a Data Scientist. Over the last one year, I have spent most of my time in learning new concepts & technologies. I am eager to see my theory shaping into work.
In my next blog post I will explain the tools, concepts, technologies, online forums available on internet.

Cluster Analysis using R

2013-12-17T15:19:00.002+04:00

In this post, I will explain you about Cluster Analysis, The process of grouping objects/individuals together in such a way that objects/individuals in one group are more similar than objects/individuals in other groups.
For example, from a ticket booking engine database identifying clients with similar booking activities and group them together (called Clusters). Later these identified clusters can be targeted for business improvement by issuing special offers, etc.
Cluster Analysis falls into Unsupervised Learning algorithms, where in Data to be analyzed will be provided to a Cluster analysis algorithm to identify hidden patterns within as shown in the figure below.

In the image above, the cluster algorithm has grouped the input data into two groups. There are 3 Popular Clustering algorithms, Hierarchical Cluster Analysis, K-Means Cluster Analysis, Two-step Cluster Analysis, of which today I will be dealing with K-Means Clustering.
Explaining k-Means Cluster Algorithm:
In K-means algorithm, k stands for the number of clusters (groups) to be formed, hence this algorithm can be used to group known number of groups within the Analyzed data.
K Means is an iterative algorithm and it has two steps. First is a Cluster Assignment Step, and second is a Move Centroid Step.
CLUSTER ASSIGNMENT STEP: In this step, we randomly chose two cluster points (red dot & green dot) and we assign each data point to one of the two cluster points whichever is closer to it. (Top part of the below image)
MOVE CENTROID STEP: In this step, we take the average of the points of all the examples in each group and move the Centroid to the new position i.e. mean position calculated. (Bottom part of the below image)
The above steps are repeated until all the data points are grouped into 2 groups and the mean of the data points at the end of Move Centroid Step doesn’t change.

By repeating the above steps the final output grouping of the input data will be obtained.

Cluster Analysis on Accidental Deaths by Natural Causes in India using R

Implementation of k-Means Cluster algorithm can readily downloaded as R Package, CLUSTER . Using the package we shall do cluster analysis of Accidents deaths in India by Natural Causes.

Steps implemented will be discussed as below:

The data for our analysis was downloaded from www.data.gov.in.

Between 2001 & 2012. Input data is displayed as below:

For any cluster analysis, all the features have to be converted into numerical & the larger values in the Year Columns are converted to z-score for better results.

Run Elbow method (code available below) is run to find the optimal number of clusters present within the data points.

Run the K-means cluster method of the R package & visualize the results as below:

Code:

#Fetch data

data= read.csv("Cluster Analysis.csv")

APStats = data[which(data$STATE == 'ANDHRA PRADESH'),]

APMale = rowSums(APStats[,4:8])

APFemale = rowSums(APStats[,9:13])

APStats[,'APMale'] = APMale

APStats[,'APFemale'] = APFemale

data = APStats[c(2,3,14,15)]

library(cluster)

library(graphics)

library(ggplot2)

#factor the categorical fields

cause = as.numeric(factor(data$CAUSE))

data$CAUSE = cause

#Z-score for Year column

z = {}

m = mean(data$Year)

sd = sd(data$Year)

year = data$Year

for(i in 1:length(data$Year)){

z[i] = (year[i] - m)/sd

}

data$Year = as.numeric(z)

#Calculating K-means - Cluster assignment & cluster group steps

cost_df <- data.frame()

for(i in 1:100){

kmeans<- kmeans(x=data, centers=i, iter.max=100)

cost_df<- rbind(cost_df, cbind(i, kmeans$tot.withinss))

}

names(cost_df) <- c("cluster", "cost")

#Elbow method to identify the idle number of Cluster

#Cost plot

ggplot(data=cost_df, aes(x=cluster, y=cost, group=1)) +

theme_bw(base_family="Garamond") +

geom_line(colour = "darkgreen") +

theme(text = element_text(size=20)) +

ggtitle("Reduction In Cost For Values of 'k'\n") +

xlab("\nClusters") +

ylab("Within-Cluster Sum of Squares\n")

clust = kmeans(data,5)

clusplot(data, clust$cluster, color=TRUE, shade=TRUE,labels=13, lines=0)

data[,'cluster'] = clust$cluster

head(data[which(data$cluster == 5),])

Fetch Twitter data using R

2013-10-24T11:58:00.001+04:00

This short post will explain how you can fetch twitter data using twitteR & StreamR packages available in R. In order to connect to twitter API, we need to undergo an authentication process known as OAuth explained in my previous post.

Twitter data can be fetched from twitter in two ways: a) Rest API b) Streaming Api.
In today's blog post we shall go through both using Rest API & Stream API.

twitteR Package:
One of the available package in R for fetching Twitter Data. The package can be obtained from here.
This package allows us to make REST API calls to twitter using the ConsumerKey & ConsumerSecret code. Code below illustrates how as how to extract the Twitter Data.
This package offers below functionality:

Authenticate with Twitter API
Fetch User timeline
User Followers
User Mentions
Search twitter
User Information
User friends information
Location based Trends
Convert JSON object to dataframes

REST API CALLS using R - twitteR package:

Register your application with twitter.
After registration, you will be getting ConsumerKey & ConsumerSecret code which needs to be used for calling twitter API.
Load TwitteR library in R environment.
Call twitter API using OAuthFactory$new() method with ConsumerKey & ConsumerSecret code as input params.
The above step will return an authorization link, which needs to be copied & pasted in the internet browser.
You will be redirected to Twitter application authentication page where you need to authenticate yourself by providing you twitter credentials.
After authenticating , we will be provided with a Authorization code, which needs to be pasted in the R console.
Call registerTwitterOAuth().

Source Code:
library(twitteR)

requestURL <- "https://api.twitter.com/oauth/request_token"

accessURL = "https://api.twitter.com/oauth/access_token"

authURL = "https://api.twitter.com/oauth/authorize"

consumerKey = "XXXXXXXXXXXX"

consumerSecret = "XXXXXXXXXXXXXXXX"

twitCred <- OAuthFactory$new(consumerKey=consumerKey,

consumerSecret=consumerSecret,

requestURL=requestURL,

accessURL=accessURL,

authURL=authURL)

download.file(url="http://curl.haxx.se/ca/cacert.pem",

destfile="cacert.pem")

twitCred$handshake(cainfo="cacert.pem")

save(list="twitCred", file="twitteR_credentials")

load("twitteR_credentials")

registerTwitterOAuth(twitCred)#Register your app with Twitter.
StreamR Package:
This package allows users to fetch twitter Data in real time by connecting to Twitter Stream API. We can obtain the package from here. Few important functions this package offers are: it allows R users to access Twitter's search streams,user streams, parse the output into data frames.

filterStream() - filterStream method opens a connection to Twitter’s Streaming API that will return public statuses that match one or more ﬁlter predicates like search keywords. Tweets can be ﬁltered by keywords, users, language, and location. The output can be saved as an object in memory or written to a text ﬁle.

parseTweets() - This function parses tweets downloaded using filterStream, sampleStream or userStream and returns a data frame.
Below code example shows how to fetch data in real time using RStream:
library(streamR)
library(twitteR)
load("twitteR_credentials") # make using the save credentials in the previous code.
registerTwitterOAuth(twitCred)
filterStream(file.name = "tweets.json", track = "#bigdata",timeout = 0, locations=c(-74,40,-73,41), oauth = twitCred)
Executing the above will capturing Tweets on "#bigdata" from "NEW YORK" location. Here when we mention timeout=0, we are setting it to fetch continuously, to fetch records for certain time then use timeout=300 (fetches data for 300 secs)
To Parse the fetched tweets use the below code:
tweets.df <- parseTweets("tweets.json")