meet Saptarsi

Questions on Java

2014-08-19T09:34:00.000-07:00

Friends , here is my first attempt , the SCJP book by Kathy has given me few questions , I would like to enrich it further , but time for a first cut

OOPS, Java

· Is Java compiled or interpreted?

· How can we declare two public classes in a Java source file?

· Is there a restriction on naming the source file?

· Give an example of code statement, where do while and while will give different output?

· What is the default access, if we do not specify any access modifiers?

· How method overloading and method overriding is different?

· Give example of a non access modifier?

· Why should some classes or methods be marked as final?

· How the abstract class is different from an interface as far as abstract methods are concerned?

· How garbage collection works in C# and Java ?

· What is boxing and Unboxing ?

· How can we implement threads ?

· What are the different states of a thread ?

· What is a web service? How this is related with SOAP, UDDI and WSDL?

· What are the different ways to maintain state in a web application , relative advantage and disadvantages ? ( Hint : Session Variable, Cookies , Hidden Fields , Query String ) . For .Net view state is important.

· What is MVC Architecture ?

· Explain 3- tier Architecture ?

· Explain Service oriented architecture

· What is the difference between default and package access modifier

· Let’s say Class A has a private method called fun() , class B extends Class A and defined fun() , what this phenomenon is called ?

· What is the use of Final Keyword ? Can we declare a variable as final ?

· Is it possible to define a abstract class , with non abstract member s?

· When should we use an arraylist in place of an array?

· What is the method that can be used to traverse an arraylist?

· What is checked and unchecked exception ? Give example of both type of exceptions.

· How to use your own exception , so let’s say you are taking age as an input and you want to throw an exception when it is less than zero, how will you achieve it ?

· Give some example of statements that should be put in finally ?

· Does Java allow multiple inheritance ? What is ‘Deadly Diamond of Death”

· Class A

{

public void fun()

{

}

Class B extend A

{

Protected void fun()

}

Is there a problem in the above code ?

· Give an example how polymorphism can be achieved using method overriding?

public int foo()

{

char c = ‘c’;

return c

}

Is this code legal ?

· What is default constructor supplied by the compiler ? Is there any caution , that you need to follow when you create your own constructor, will the default constructor be supplied in the following case

class Horse()

{

void Horse ()

}

· Which variables are in stack , which are stored in heap ?

· What do we mean by saying strings are immutable objects ? what is role of ‘String Constant Pool” . Which classed do we use to overcome the limitations of string ?

Beyond bars, pie charts and histograms …………

2014-07-28T21:36:00.001-07:00

( Was written for the Institute of Engineering and Management 2014 Magazine , put online for better access , specially for my students )

The reference of the popular saying “A picture is worth a thousand words” goes back to early part of 20^th century. If we take graphs as a rough equivalent to picture, apart from bringing the quick interpretability factor, the other benefit of the same is a summarized view of the data. As a case in point, let’s look at the below crime data of US [U.S. Department of Justice]. I am taking crime data in property subdivided in Burglary, Larceny theft and Motor Vehicle theft (For those of you, who have been wondering what is larceny like me, it’s just a legal term for stealing). The numbers indicate rates per 100,000 populations.

States	Burglary	Larceny- theft	Motor Vehicle Theft	Total
Alabama	953.8	2650	288.3	3892.1
Alaska	622.5	2599.1	391	3612.5
Arizona	948.4	2965.2	924.4	4838
Arkansas	1084.6	2711.2	262.1	4057.9
California	693.3	1916.5	712.8	3322.6
Colorado	744.8	2735.2	559.5	4039.5
Connecticut	437.1	1824.1	296.8	2558
Delaware	688.9	2144	278.5	3111.4
Florida	926.3	2658.3	423.3	4007.9
Georgia	931	2751.1	490.2	4172.3

Many of you would apply your ‘CAT – cracking ´ DI skill and would do some quick mental math way better than me, but there is no denying, the below chart reveals much more to mere mortals. We are very quick to spot which are two top/bottom crime prone states.

Let’s look at another view of the same data, this time a stacked bar chart

A stacked bar chart takes contribution of each state as 100% and then depicts the contribution from different components with different color portions. It will be difficult to miss

Ø Alabama and Arkansas have a relatively lesser proportion of motor vehicles theft

Ø Whereas Alaska and Colorado has relatively low proportion of burglary

I hope that now some of you might have been convinced on the usefulness of traditional charts. So this is an opportune time for “kahanime Twist”. There is a trite ‘Gyan’ available specially for negotiating on price while shopping from hawkers. You should not start negotiating on the thing you liked and want to purchase, rather do on something else nearby (for a lack of an original example let me settle for if you are trying to buy a রুমাল (handkerchief) ask the price of a বেড়াল (cat)), bargain hard and then casually bring him to the piece you actually want to buy. Because if the hawker understands you really fell for it, the chance of negotiation is bleak. So with this entire prologue, let me just say gone are the days, where we can impress people with graphical reports with bar charts, histograms, pie charts, line charts etc. This can be felt more with the advent of “Big Data”.

Commercially ‘Big Data” is differentiated by three Vs, Volume, Variety and Velocity. Without going into too much of details, I guess it will be sufficient to draw your attention to the number of tweets per day, which is 500 millions! This gave birth to a new term named Tweets per second (TPS). For understanding variety we do not need to look beyond the photos, blogs, chats, emails, medical images, sensor logs, RFID logs generated at each second. When we talk about velocity we are not talking about the rate data is getting generated, but also the rate at which it is becoming obsolete. One practical example of the same is Gurgaon police now a days have added tweets in their dashboard. So if people are tweeting on some quarrel breaking out , bedlam or some other reason a pandemonium , the police can exactly track the area at real time via the tweets, so they can send the nearest petrol car within minutes. So contrary to the”filmy” style of arriving well after when crime is done and criminals fled, now the cops arrive well in time.

There has been a toolbox of visualization techniques on this new “avatar” of the Data and a lot can be written on them. I will choose just one of them as a class representative (Class of new visualization techniques)

So meet “Mr Word Cloud”. You might not have known his formal name but surely you had a chance or deliberate tryst with him somewhere in your internet life. Though he is not a toddler, but at least he is your age group (1992 born). So here is a word cloud used by a leading data analyst in his blogs emphasizing on the importance of data. It is very handy to summarize large body of texts, blogs, and tweets. Oh by the way I was forgetting, he does not mind being called ‘tag cloud’ as well.

It’s easy to note that words have different font size, which is proportional to their frequency aka importance, the vertical and horizontal orientation does not mean much.

Roughly there are two ways you can start using word clouds.

1. Use a site like http://www.wordle.net/ . Either copy paste your text or give the url of a site which has a Atom or RSS feed (very crudely a mechanism to exchange dynamically updated information).

2. Use a computing environment like R ( it’s one of the most used open source)statistical and data analysis computing environment. (http://cran.r-project.org/)

The below one is freshly baked for you, using not more than 10 lines of code in R from few of the movie reviews (NDTV, Hindustan Times, TOI etc. ) on a movie and no prize for guessing the name of the movie ( I used some filtering of selecting only words which has appeared at least 3 times and some further work can be done in removing ‘stop words’ which are film industry specific, basically words which are likely to be present in any movie review , irrespective of the movie like movie, film etc. Without watching the movie¸ I would not be probably wrong in concluding if the movie is to be watched it’s for Aamir.

I hope to make a case for visualization techniques and did you guys think it can help hide the author’s excellent writing prowess or lack of it; well none of your provocations will elicit an answer from me.

Exploratory data analysis on P/E ratio of Indian Stocks

2014-03-31T11:50:00.000-07:00

Price Earnings ratio (P/E) is one of the very popular ratios reported with all stocks. Very simply this is thought as - Current Market Price / Earning per Share. An operational definition of Earning per Share would be Total profit divided by # of Shares . I will redirect interested readers for further reading to

www.investopedia.com/terms/p/price-earningsratio.asp

In this post, I would just like to show, how we can grab P/E data from Web and create some visualizations on it. My focus right now is Indian stocks and I intend to use the below website

http://www.indiainfoline.com/MarketStatistics/PE-Ratios/

So my first step is gearing up for the data extraction and essentially that is the most non-trivial task. As shown in the figure below, there is separate pages for each sector and we need to click on individual links , to go to that page and get the P/E ratios.

Here is something , I did outside ‘r’ , creating a csv file with the sector names , using delimiters while importing text and paste special as transpose , here is how my csv file would look. I would never discourage using multiple tools as this would be required to solve real world issues

So now I can import this in a dataset and read one row at a time and go to necessary URLs , but god have different plans J , it’s not that straightforward

Case 1 : Single word sector names :

We have sector as ‘Banks’and the sector link is as below

http://www.indiainfoline.com/MarketStatistics/PE-Ratios/Banks-Sector

Again it is a no brainer , we can pick up the base url , append the sector name after a forward slash and then append the string ‘-Sector’ , this is true for most single word sector names like ‘FMCG’ , ‘Tyres’ , ‘Heathcare’ etc

Case 2: Multiple words without ‘-‘ , ‘&’ and ‘/’

We have sector as ‘Tobacco Products’ and the sector link is as below

http://www.indiainfoline.com/MarketStatistics/PE-Ratios/Tobacco-Products-Sector

This is also not that difficult apart from adding the ‘-Sector’ we need replace the spaces by a ‘-‘ .

Case 3: Multiple words with a ‘-‘

We have sector name as ‘IT-Software’, where we have to remove other spaces if exiting. There can be several other cases, but for discussion sake , I will limit myself here

Case 4: Multiple words with a ‘/‘

We have sector name as ‘Stock/ Commodity Brokers’, so the “/” needs to be removed

# Reading in dataset

sectorsv1 <- read.csv("C:/Users/user/Desktop/Datasets/sectorsv1.csv")

# Converting to a matrix , this is a practice generally I follow

sectorvm<-as.matrix(sectorsv1)

we can access individual sectors by , sectorvm[rowno,colon]

pe<-c()

cname<-c()

cnt<-0

baseurl<-'http://www.indiainfoline.com/MarketStatistics/PE-Ratios/'

sectorvm<-as.matrix(sectorsv1)

for(i in 1:nrow(sectorvm))

{

securl<-sectorvm[i,1]

# Fixed true indicated the string is to matched as is and is not a regular expression

# Substitution of the different cases as we explained , we will point out using gsub instead of sub

# else only the first instance will be replaced

if(length(grep(' ',securl,fixed=TRUE))!=1)

{

securl<-paste(securl,'-Sector', sep="")

}

else

{

securl<-gsub(' ', '-', securl, ignore.case =FALSE, fixed=TRUE)

if(length(grep('---',securl,fixed=TRUE))==1)

{

securl<-gsub(' ---', '-', securl, ignore.case =FALSE, fixed=TRUE)

}

if(length(grep('&',securl,fixed=TRUE))==1)

{

securl<-gsub('&', 'and', securl, ignore.case =FALSE, fixed=TRUE)

}

if(length(grep('/',securl,fixed=TRUE))==1)

{

securl<-gsub('/', '', securl, ignore.case =FALSE, fixed=TRUE)

}

if(length(grep(',',securl,fixed=TRUE))==1)

{

securl<-gsub(',', '', securl, ignore.case =FALSE, fixed=TRUE)

}

securl<-paste(securl,'-Sector', sep="")

}

fullurl<-paste(baseurl,securl, sep="")

print(fullurl)

if (url.exists(fullurl))

{

petbls<-readHTMLTable(fullurl)

# Exploring the tables we found out relevant information on table 2

# Also the data is getting stored as factor , just doing an as.numeric will not suffice

# we need to do an as.character and then an as.numeric

pe<-c(pe,as.numeric(as.character(petbls[[2]]$PE)))

cname<-c(cname, as.character (petbls[[2]]$Company))

cnt = cnt + 1

}

Different functions that we have used are explained as below

readHTMLTables -> Given a url , this function can retrieve the contents of the <Table> tag from html page. We need to use appropriate no. for the same. Like in this case we have used table no 2.

Grep, Paste, Gsub are normal string functions, grep finds occurrence of a string in another, paste concatenates and gsub does the act of replacing.

As.numeric(as.character()) had a lasting impressing on my mind as an innocuous and intuitive as.numeric would have left me only with the ranks.

url.exists :-> it is a good idea , to check the existence of the url , given we are dynamically forming the URLs.

Now playing with summary statistics:

We use the describe function from psych package

n	mean	sd	median	trimmed	mad	min	max	range	skew	kurtosis	se
1797	59.71	76.92	20.09	46.64	29.79	0	587.5	587.5	2.15	7.25	1.81

hist(pe,col='blue',main='P/E Distribution')

We get the below histogram for the P/E ratio , which shows it is nowhere near a normal distribution , with it’s peakedness and skew as confirmed from the summary statistics as well

We will never the less do a normalty test

shapiro.test(pe)

        Shapiro-Wilk normality test

data:  pe

W = 0.7496, p-value < 2.2e-16

Basically the null hypothesis is, the values come from a normal distribution and we see the p value to be very insignificant and hence we can easily reject the null.

Drawing a box plot on the P/E ratios

boxplot(pe,col='blue')

Finding the outliers

boxplot.stats(pe)$out

484.33 327.91 587.50

cname[which(pe %in% boxplot.stats(pe)$out)]

[1] "Bajaj Electrical" "BF Utilities"     "Ruchi Infrastr."

Of course no prize guessing we should stay out of these stocks

So if we summarize this is kind of exploratory data analysis on PE ratio of Indian stocks

· We saw, we can get content out of url and html tables

· We added them in a data frame

· Looked at summary statistics , histogram and did a normality test

· Plotted a box plot and found the outliers

happiness equation !!!!

2013-12-19T10:05:00.003-08:00

Who knew that happiness can have a equation, I was glancing through one of the lectures by Swami Sarvapriyananda, he introduced to something called as a happiness equation , something proposed by Martin Seligman, the father of positive psychology. The version he shared and which is in perfect agreement with the Purusartha’s of Vedanta is something like this

H = P + E + M

P is for pleasure. Good food, movies . they are short-lived.
E comes from engagement, our profession, creativity , research
M is for meaningful, for others. First beyond you and then beyond your family.

Happiness from E and M should make up for the most. u dine at a good restaurant , u spend 1 hour teaching a poor student. after some months , years the later activity will give you much more happiness.

Very profound!!!

More can be watched @ goo.gl/n5TlWx

Statistica , very few points

2013-11-11T09:34:00.001-08:00

Don’t think I am qualified, but got a chance to interact with a professional from Statistica and thought of sharing few things about the tool

· It has amazing integration with ms suites , may be something in the cards

· Import is easy and wizard driven , supports multitude of data files ,allows connection with a file as well as database

· It also allows to work on data from multiple sources

· It has good statistical capabilities

· Did not get a chance to look at all functionality

· Correlation and Regression looked good with linear, multiple, factor regression

· Help files are good and comes with quite a few sample datasets

· To me a winner is , it’s vba coding interface , which will make life simpler

· You do not necessarily , need to bring all data locally and process , there is a technique available for the same as well

I know , this is tip of the ice berg , but just thought of sharing what ever I gathered, will keep you posted.

Classification using neural net in r

2013-10-09T11:40:00.000-07:00

This is mostly for my students and myself for future reference.

Classification is a supervised task , where we need preclassified data and then on new data , I can predict.
Generally we holdout a % from the data available for testing and we call them training and testing data respectively. So it's like this , if we know which emails are spam , then only using classification we can predict the emails as spam.

I used the dataset http://archive.ics.uci.edu/ml/datasets/seeds# . The data set has 7 real valued attributes and 1 for predicting . http://www.jeffheaton.com/2013/06/basic-classification-in-r-neural-networks-and-support-vector-machines/ has influenced many of the writing , probably I am making it more obvious.

The library to be used is library(nnet) , below are the list of commands for your reference

1. Read from dataset

seeds<-read.csv('seeds.csv',header=T)

2. Setting training set index , 210 is the dataset size, 147 is 70 % of that

   seedstrain<- sample(1:210,147)

3. Setting test set index

   seedstest <- setdiff(1:210,seedstrain)

4. Normalize the value to be predicted , use that attribute of the dataset , that you want to predict

   ideal <- class.ind(seeds$Class)

5. Train the model, -8 because you want to leave out the class attribute , the dataset had a total of 8 attributes with the last one as the predicted one

   seedsANN = nnet(seeds[seedstrain,-8], ideal[seedstrain,], size=10, softmax=TRUE)

6. Predict on training set

   predict(seedsANN, seeds[seedstrain,-8], type="class")

7. Calculate Classification accuracy

   table(predict(seedsANN, seeds[seedstest,-8], type="class"),seeds[seedstest,]$Class)

Happy Coding !

DBMS : few questions for freshers

2013-09-04T11:00:00.000-07:00

Hello , once you are ready with the 'HR' type of questions , (you can take a look at some of the questions and answers at HR questions) it is obviously important for getting ready for the technical and one of the subjects , and DBMS is one of the leading ones from both the parties. So I pen down few DBMS questions, this are all indicative ones , just to give you an idea on the depth and breadth. Normalization, Transactions , SQL and Indexing , I have seen to be all time favorites.

Feel free to give your comments , post answers , more questions , any other feedback.

Imagining you guyz are reading this because of your impending campus , read smartly , fine balancing with enjoying life , work on a plan and you will be partying in time :)

DBMS :

General

What is OLAP and OLTP?
What are the advantages of DBMS over file system? ( Do not forget key ones like transaction , normalization, indexing , locking , logging etc. )
Why RDBMS called RDBMS ? Stress on the relational part.
Draw an ER Diagram for the project you did ?
.What is the degree of a relationship? Does relationships always needs a table ?
How do you achieve generalization and specialization in an ER?

Normalization

Why do we normalize?
What are the different anomalies?
Why do we denormalize ?
Give an example of a table , which is not in 3 NF , explain diffrent anomalies in that context , tell how you will make it normalized

Transaction

What is a transaction?
What is ACID property? Explain each one with example.
How the durability property is implemented ?
How do you implement a transaction from a programming language like C# or java?
Can a transaction be partially committed ?
What is locking , what is two phase locking ?
What is seralizability ?

Integrity

What are different type of constraints ?
What is primary key , foreign key , candidate key ?
What is unique , Check constrains
Difference between a primary key and unique constraint ?
Is it possible to create a table with out a primary key ?

Indexing:

Why do we use index?
What are the overheads of index?
What is the difference between clustered index and a non-clustered index ?
What is the difference between B Tree and B+ Tree
Another way of asking the same question , what are the diffrent data structures that are used

SQL:

What is difference between a function and stored procedure ?
What is cursor ? What are the different types of cursor in oracle ?
What are the different indices that oracle support?’ ( Special focus on bitmap indices)
What is the difference on where and having clause ?
Questions on NULL
What is the difference between char data type and varchar data type?
What is a view? What is materialised view? Why they are used?What are different types of joins , what is difference between a Cartesian product and a full outer join ( Practise this with few examples )
What is the function of UNION ? Is it different from UNION ALL?
What is DDL and DML, how truncate is different from Delete
Where will you use trigger , what are the different types of trigger ?
Why do we create packages ?
How do we handle exceptions?
What is the role of dual ?
If two tables have PK - FK relationship and you want when the PK gets deleted , the FK entries also gets deleted , how do you do that ?
Why and how do you create sequences?
How can you delete dupicate data from a table ?

Preparing for Campus ...

2013-08-18T09:27:00.002-07:00

The most important thing in a campus interview is to be yourself. It is easier said than done as in all probability it is your first job interview and you are in cloths, which you generally do not wear, additionally there is a tie hanging awkwardly. To top it, the procedure has started from morning, so it is quite exhausting mentally and physically. So best thing is be prepared , I will give you simple dos and don’ts from the experience that I have during my years in Cognizant and various student interactions in Calcutta University and Institute of Engineering and Management ( The views are completely personal)

Ø Read each and every word of your resume , don’t keep anything if you are not comfortable

Ø Introduce yourself is a sure question , don’t fumble on this , set it up here , tell about your subject interest , hobbies , if you are a student partner , a class representative , some uncanny hobbies, part of your college computer society . If you have done something academically very good don’t forget to mention that.

Ø Strength and weakness is optional and if you say something, please be prepared to back it up. So if you say you have analytic mind or you are hard working, you should be able to give few examples

Ø You need not be too forthcoming on your weaknesses, and please do not mention like I am too emotional or I am loose tempered.

Ø Have a professional looking email id , not something like itwasnoteasy , walkthetalk , or futufutejyosthna . Don’t laugh , I have faced it

Ø I will say write it down and practice, record and listen to if required.

Ø Don’t over commit on your subjects , two theoretical subjects , one/two programming language , one database should be sufficient

Ø Prepare well on your final year / Internship project , architecture , business sense / unique proposition should come out very clearly

Ø People have told their hobby is reading, when I have asked what book they last read, they are drawing a blank.

Ø Be prepared for questions like, “Why do you want to join TCS”, “Why should we hire you”. I would have been honest with these questions like “I have heard very positive things from my seniors, and Tata name has a huge brand value. The training programs are excellent ……..”. Not something like this is my dream company and I have always thought of joining this company and all that.

Ø If you have a year lag or considerable % drop please prepare for the same question.

Ø On technical part , please understand people who are coming will be 10 + years experience and mostly they might not ask you definitions , I will ask you to focus from interview perspective , not an examination purpose , know the examples , understand the concepts , discuss with friends to gain confidence.

Ø Pseudo code will suffice most of the cases , so do not spend lot of time on syntax

For few technical questions on DBMS , you may visit DBMS Questions

Wish you All the very best.

How ACIDic are Transactions really!

2012-12-21T08:46:00.000-08:00

I got a little bit distracted while I was hearing about the ‘No SQL Databases’ , one of the key characteristics is they do not support Full ACID transactions, This brings back some of the memories. Not all of them pleasant, this is/was one of the sure shot menus to be served by the interviewers, be it job interviews or college viva. Honestly, now I think the entire ‘ACID’ thing was little overdone. I have no disillusion that it is one of the widely written topics , a google search below brings 1.5 billon search pages, but I thought of making a point, may be at the cost of being repetitive.

I talk very briefly , highlighting key concepts.

ACID (Atomicity, Consistency, Isolation, Durability)

Atomicity: Either all or none. Reverts to earlier state in case of a failure (system, hardware, software) . Handled by Recivery-management component

Consistency: When the transaction is complete it is in a consistent state. Responsibility of an Application Programmer.

Durability: Once a transaction is completed, the effects are permanent. No failure can change the same. This is responsibility of recovery-management component

Isolation: Concept of Serializability, responsibility of concurrency control

Read it again please carefully, is not concurrency and recovery all that we are talking. Keeping consistency is like having commonsense, which is uncommonJ.

I think we should talk about CR Properties rather than ACID.

Opting for shorter movies, be aware u might be cutting the entertainment too!

2012-11-23T09:13:00.000-08:00

Hello Friends,

This time I thought to bring in little more spice and thought of focusing on movies. I don’t know about you but I am a movie buff. Often on a weekend when I am trying to pick up a movie from my movie repository, which spans to some TBs now, I feel little lost. Apart from a general rating or a perception, the length of the movie plays a role in the choice, simple reason; the movie needs to be cramped between other demanding priorities.

So last Saturday, when I was in between this process, and I was searching for a movie less than 1 hour 30 minutes (There was a hard stop on that) my wife commented but “The short movies are generally not so good”. I did not pay much heed to that then (Don’t conclude anything from this please), but later on I thought hold on, is that a hypothesis? Can I do something statistically here? And here we are. We will talk little bit on correlation, normal distribution etc. I use ‘R’, but it is so simple , we can even use excel sheet do the same.

Correlation:

This is an indicator whose value is between -1 and 1 and it indicates strength of linear relationship between two variables. Leave the jargon, many cases we relate features. The typical law of physics like speed and displacement may have a perfect correlation, but those are not the point of interest. However a point of interest may be is there a relation between say

a) IQ Score of a person and Salary drawn

b) No. of obese people in an area vis-à-vis no. of fast-food centers in the locality

c) No. of Facebook friends , with relationship shelf life

d) No. of hours spent in office and attrition rate for and organization

An underlying technicality, I must point out here is both of the variables should follow a normal distribution.

Normal Distribution:

This is the most common probability distribution function, which is a bell shaped curve, with equal spread in both side of the mean. Associate to manager alike, you must have heard about normalization and bell curve while you face/do the appraisal. Most of the random events across disciplines follow normal distribution. The below is an internet image.

So I picked up movie information and like any one of us picked it up from IMDB (http://www.imdb.com/) and I put it in a structured form like the below, the ones highlighted below may not be required at this point of time, I kept it just for some future work in mind. The list was prepared manually; I will keep on hunting for some API and all and would keep you posted on the same.

Name	Year of Release	Rating	Duration	Small Desc
Skyfall	2012	8.1	143	Bond's loyalty to M is tested as her past comes back to haunt her. As MI6 comes under attack, 007 must track down and destroy the threat, no matter how personal the cost.

At this point of time I have taken 183 movies. I have stored it as a csv file.

First thing first, there are various formal ways to test whether it follows a normal distribution, I would just plot histograms and see how this looks like, both the variable seem to follow normal distributions closely.

Below are the commands for a quick reference. What I just adore about R is it’s simplicity, with just so few commands we are done

film<-read.csv("film.csv",header=T)# Reading the file in a list object

x<-as.matrix(film) # Converting the list to a matrix, for histogram plotting

y<-as.numeric(x[,3]) # Converting the movie rating to a numeric vector

y<-as.numeric(x[,4]) # Converting the movie duration to a numeric vector

hist(y,col="green",border="black",xlab="Duarion",ylab="mvfreq",main="Mv Duration Distribution",breaks=7)

hist(y,col="blue",border="black",xlab="mvRtng",ylab="mvfreq",main="Mv Rtng Distribution",breaks=9)

cor(y,z) # Calculate Correlation Coefficient between rating and duration

Interestingly the correlation turns out to be .48 in this case, which says there is a positive correlation between this two phenomenon and the correlation is not small. We can set up a hypothesis “ There is no correlation “ and a level of significance and test the hypothesis. However .48 is a high value and I am sure we would reject the hypothesis that there is no correlation.

So someway or other the rating goes up with the duration of the movie.

I leave it to you for interpretation, but next time you might look at the movie duration for taking a call ! Mr. Directors , it might be a tips for you who knows and may be to me wify is always right. May be all that short is not that sweet.

With that I will call it a day, hope you enjoyed reading. I will be coming on with more such Looking forward to your feedbacks and comments

SPSS Modeler a quick intro

2012-11-10T05:05:00.000-08:00

Hello Friends

It’s time I thought of talking to you again. I happened to have some exposure to SPSS modeler and wanted to share some of the information with you. There are several articles, blogs, demos and product manuals of the same to guide/confuse you. I would just share a gist. Here’s how I would go. I would start with exploring the IBM stack in this space and then give you tiny tit-bits on the modeler. Again when I explore the stack it’s based on my individual perception and might not align with the formal positioning. Most of the products are individually licensed. We will have to mix and match while offering for customer based on their need of operational and analytical decision management. Currently I am limiting to the structured data space and am not touching upon Social Network, Text Analytics or products like Big Insight or Watson which envisages AI to be taken to a crescendo and have capabilities to make many highly paid business consultants redundant.

Product Portfolio:

· SPSS Modeler desktop :It’s a GUI based cool tool , with blocks available for different tasks, where you can define your data mining , predictive analytics tasks. The desktop version can really commoditize data mining with its drag and drop features. Suited for data scientists as well as business users who have a flare to the algorithms.

· SPSS Modeler Server edition would be required for scheduling of jobs, batch mode execution etc

· CADS will be required for collaboration, deployment and scoring. Collaboration will allow multiple people participate in model building and scoring will allow real time integration, with exposing the models as web services. So for a real time analytical application like fraud detection or day to day web experiences like association for cross sell and up sell it’s surely needed. Its champion, challenger model sounds fresh, where the challenger models become new champion on better result.

· SPSS Statistica : Offering much more flexibility with scripting and customization. More suited for techies and statisticians. Can complement modeler for advanced statistical tasks.

· Analytical Decision Management: This is for the business users. They can lay out the skeleton of the models here and the tech team can work in the background to put flesh and bone. This allows combining both business rules defined in the analytical decision management and rules coming out of SPSS modeler. It uses CPLEX which is a constraint based optimizer.

· Entity Analytics: As the name suggests this is aligned for identifying logical duplicates and de-duplication. Results of this can significantly improve modelers’ accuracy.

SPSS Modeler:

Again in this section after talking about the nodes, I only talk about some of the features I liked

- It provides you an interface to follow crisp DM methodology, which is a cross industry standard consisting of stages like (Business Understanding, Data Understanding, Data Preparation, Modeling, Evaluation and deployment)

- The data mining workflow will be defined in terms of various nodes like

o Source node : For reading the data

o Record node: It affects the no. of records. It can be as simple as filtering or aggregation.

o Field node: Used for data transformations, cleaning and preparation. Automated data preparation (ADP) is a very handy filter note, allowing many easy and custom transformations with a significant probability of improving the accuracy. Anonymize allows to suppress/mask private information which is very relevant given so many prevalent compliance norms.

o Graph Node: Allows many types of visualization as well as evaluation of the models.

o Modeling Node : This is the cream, which will have the data mining models. There is another group within the modeling node which is statistical node, which yet again offers useful functionalities like PCA, Factor, discriminant analysis etc.

o Output Node : This will be required for analyzing the results

o Export Node: Allows data to be transported to other software tools like excel, SAS etc.

o Super Node: Allows grouping of multiple nodes in more reusable and modular fashion.

- Offers quite a few standard algorithms for common tasks like classification, regression, clustering, time series & association.

- Auto classifier, auto clusters really makes evaluation of models so easy. To clarify little bit more, we can use a classifier to detect a risky loan, we can be in two minds so as to which algorithms to be picked is it neural net or decision tree or may be logistic regression. Auto classifier can do an evaluation on all of them and pick the best.

- SQL Pushback: Allows to push back some of the computations to the database.

- In Database Mining : Allows SPSS Modeller , leverage native algorithms of other database vendors offering data mining capabilities like IBM Netezza, IBM DB2 InfoSphere Warehouse, Oracle Data Miner, and Microsoft Analysis Services

Overall modeler is a great tool which is easy to use and is intuitive and IBM has a rich portfolio of advanced analytics and decision management products, however so wide range may be confusing to the end customer and industry specific packaged solutions with combinations of products can demystify the same. Also packaging and readily available blocks are so far so good but need for deep domain knowledge, statistical understanding is here to stay, for superior results

I intentionally wanted to keep it short and just tickle your curiosities. The festival of lights is nearing. Wish you and your family a safe and joyous time!

Will meet you again real soon! I hope you enjoyed.

Venturing in text mining with 'R'

2012-10-19T11:07:00.000-07:00

Background:

Hello Friends, hope all of you are doing just great. I decided to create my footprint in the blog space, it comes from my desire to share few very basic steps of text mining, with all of you. I am neither a nerd, or statistician or an established data scientist and if you are one of them well, this blog is surely not for you.

I really struggled while I was experimenting with this simple stuff and thought of sharing this with each one of you. I have spent last 6-7 years as a DW & BI professional. I have seen full cycle from data extraction to information delivery, using various tools and technologies and when we talk about advanced analytics it includes advanced data mining/machine learning techniques apart from traditional OLAP. Predictive analytics will be more relevant with the splurge of data. However I see, lot of my colleagues, across organization finds themselves little awkward, out-of-place while there are talks about dimensionality reduction, Customer Segmentation, Tag Clouding, Anomaly detection. It is an acknowledged fact that data mining for both structured and unstructured data needs to be much more commoditized in the DWBI Community. I write to address this gap! Instead of a bookish bottom-up, I go top-down, with focus on a small yet intuitive task. Again it is not a ‘How to”, so I omit obvious screenshots mostly and try to bring a interactive feel in the narrative. As I said , in case you are an erudite in this field, venturing further is at your on will and risk, if you are interested about ‘R’ and how this can be used for text mining, I am trying to pen as lucidly as possible, let’s take the plunge together. So below is the flow, I will describe few terms in my own way and then talk about a simple text mining task.

Well, it’s an open source tool for statistical & visualization tasks. Formally it is positioned as an environment for Computing and Graphics, it is a successor of S which was developed in Bell labs and it is a part of the GNU. My inquisitive friends can look at http://cran.r-project.org/ and be further enlightened. Again informally it is a lightweight convenient tool which is in fact free, and is robust with lot of features. There are lots of people in different communities, discussion forums and mailing lists . You don’t need Unix and all, runs smoothly on our own “Windows”. So we can get started with ‘R’ for basic data mining and text mining jobs with any further ado.

Stemming:

I will start with example. Let me put it this way when we are doing a text analysis / mining/ natural language processing or any other kind of task where we deal with words, a basic thing would be looking at distinct words there counts for sure. So we need to count ‘run’, ‘ran’, ‘running’, ‘runner’ as one word rather than four. So we count the stem of the word rather than the form.

Stop words:

These are frequently used words, which are generally filtered out before a processing task. This is an absolute everyday phenomenon we encounter every day. Stop words can change depending on the nature of the text mining task.

Term Document Matrix:

Don’t scoff at me, if I say text documents are nothing but a very high dimensional vector, and lot of the dimensions are sparsely populated. Texts are high dimensional, simple reason being the words work as dimensions and we know how large than the dimensions can be. Number of dimensions can be as large as number of words in a dictionary. Well there is a popular abstraction of a text document. We as if start by extracting words one by one from a document and dumping them in a bag. So we lose the sense of grammar and order of words, but this is still a fairly useful abstraction. Loosely word and term can be used interchangeably. Only that we might have removed stop words and used stemming, to prune the word list and the final list can be significantly shorter than the original words list.

Coming back to term document matrix in the simplest form, this will have the terms and the frequency against each document. I have taken two very short documents below and then illustrated the term document matrix. The weights in this case are simple term frequency. However a popular method of weighing is combining the inverse document frequency (IDF) with the same, which takes the rarity of the word also into consideration. I will keep this for subsequent articles may be.

Document1: Text mining is cool.

Document2: This is the second text document, with almost no text.

Docs	Almost	cool	document	mining	second	text	The	this	with
1	0	1	0	1	0	1	0	0	0
2	1	0	1	0	1	2	1	1	1

R Installation and getting started:

Well this can be installed from the link I shared earlier. Follow the version for your OS. I will presume it is for windows and continue, by default setting you would get a shortcut at the desktop. You click on the same and R gets launched

Text Mining Specific Installation:

You would need to install 2 packages one TM and another Snowball which does not come with default setting. Go to packages and click on install packages. Select a CRAN Mirror; preferably select one which is geographically closer. Select the packages one by one. Installation happens automatically, we might need to load the packages.

About the task:

You can pick up any task that you want to use the default one as explained in the text mining document “Introduction to the tm Package” or “Text Mining Infrastructure in R”. The second one is a very detailed one, for interested folks, this is definitely must read. However I thought of making it little different and may be more interesting. I thought of identifying the buzz words/ trends from ‘C’ Levels of IT Offshore based service companies and I thought of choosing N. Chandrasekhar (CEO, TCS) , Francisco D’Souza (CEO, Cognizant ) and S.D.Shibulal ( CEO, Infosys) . I collected a total of 5 interviews and saved them as 5 text files. For a broad based trend surely we would need much more documents, but still this can give a decent start.

Step 1: Saved the text files in C:<>\Documents\R\win-library\2.15\tm\texts. The path will also depend on the installation options though. I created a folder interviews specific for this and saved the files here. I set the path for tm here

“Intrvw <- system.file("texts", "Interview", package = "tm")”

Step 2: I create a Corpus named “IntrvwC” with these documents

“IntrvwC<-Corpus(DirSource(Intrvw), readerControl = list(reader = readPlain, language = "eng"))”

Step 3: Strip whitespaces from the Corpora

“IntrvwC <- tm_map(IntrvwC, stripWhitespace)”

Step 4: make all the words to lower cases

“IntrvwC <- tm_map(IntrvwC, tolower)”

Step 5: Remove stop-words

“IntrvwC <- tm_map(IntrvwC, removeWords, stopwords("english"))”

Step 6: Remove punctuation

“IntrvwC <- tm_map(IntrvwC, removePunctuation)”

Step 7: Stemming the words

“IntrvwC <- tm_map(IntrvwC, stemDocument)”

Now we are done with our required task and we can look at the document term matrix for this corpora.

The document term matrix

If we just give the name of the document term matrix at the default R prompt, it will give the below result

A document-term matrix (5 documents, 602 terms)

Non-/sparse entries: 751/2259

Sparsity : 75%

Maximal term length: 41

Weighting : term frequency (tf)

Finding frequent terms:

We use the below command

findFreqTerms(dtmIntrvw, 5)

This will identify all the terms that have occurred more than 5 times in the corpora

[1] "business" "cash" "cent" "chandrasekaran" "clients" "companies" "company"

[8] "customers" "discretionary" "don<U+393C><U+3E32>t" "europe" "financial" "growth" "industry"

[15] "infosys" "insurance" "look" "margins" "opportunities" "quarter" "services"

[22] "shibulal" "spend" "spending" "strategic" "tcs" "technology" "time"

It will be audacious to conclude anything from corpora of five documents. Never the less Europe seems to be in any leadership’s mind none the less.

With that I will sign-off thanks for bearing with me. I would be more than looking forward for your comments. Wish you a happy festive time ahead with your family and friend.