Bzst...

Online data collection

2010-02-22T16:30:00.000-08:00

Online data are a huge resources for research as well as in practice. Although it is often tempting to "scrape everything" using technologies like web-crawling, it is extremely important to keep the...

Over-fitting analogies

2010-02-11T19:06:00.000-08:00

To explain the danger of model over-fitting in prediction to data mining newcomers, I often use the following analogy:Say you are at the tailor's, who will be sewing an expensive suit (or dress) for...

Drag-and-drop data mining software for the classroom

2010-01-26T17:14:00.000-08:00

The drag-and-drop (D&D) concept in data mining tools is very neat. You "drag" icons (aka "nodes") that do different operations, and "connect" them to create a data mining process. This is also...

Creating map charts

2010-01-06T06:00:00.001-08:00

With the growing amount of available geographical data, it is useful to be able to visualize one's data on top of a map. Visualizing numeric and/or categorical information on top of a map is called a...

My newest batch of graduating data mining MBAs

2009-12-22T18:48:00.001-08:00

Congratulations to our Smith School's Fall 2009 "Data Mining for Business" students. I look forward to hearing about your future endeavors -- use data mining to do good!

Stratified sampling: why and how?

2009-12-12T07:28:00.001-08:00

In surveys and polls it is common to use stratified sampling. Stratified sampling is also used in data mining, when drawing a sample from a database (for the purpose of model building). This post...

The value of p-values: Science magazine asks

2009-11-06T19:09:00.001-08:00

My students know how I cringe when I am forced to teach them p-values. I have always felt that their meaning is hard to grasp, and hence they are mostly abused when used by non-statisticians. This is...

Testing directional hypotheses: p-values can bite

2009-10-26T19:32:00.001-07:00

I've recently had interesting discussions with colleagues in Information Systems regarding testing directional hypotheses. Following their request, I'm posting about this apparently illusive issue....

SAS On Demand: Enterprise Miner -- Update

2009-10-08T13:49:00.001-07:00

Following up on my previous posting about using SAS Enterprise Minder via the On Demand platform: From continued communication with experts at SAS, it turns out that with the EM version 5.3, which is...

SAS On Demand: Enterprise Miner

2009-10-02T15:48:00.001-07:00

I am in the process of trying out SAS Enterprise Miner via the (relatively new) SAS On Demand for Academics. In our MBA data mining course at Smith, we introduce SAS EM. In the early days, we'd get...

Interpreting log-transformed variables in linear regression

2009-09-14T18:28:00.001-07:00

Statisticians love variable transformations. log-em, square-em, square-root-em, or even use the all-encompassing Box-Cox transformation, and voilla: you get variables that are "better behaved". Good...

Creating color-coded scatterplots in Excel: a nightmare

2009-08-30T20:04:00.001-07:00

Scatterplots are extremely popular and useful graphical displays for examining the relationship between two numeric variables. They get even better when we add the use of color/hue and shape to...

Data Exploration Celebration: The ENBIS 2009 Challenge

2009-08-20T10:23:00.001-07:00

The European Network for Business and Industrial Statistics (ENBIS) has released the 2009 ENBIS Challenge. The challenge this time is to use an exploratory data analysis (EDA) tool to answer a bunch...

Histograms in Excel

2009-06-13T04:40:00.001-07:00

Histograms are very useful charts for displaying the distribution of a numerical measurement. The idea is to bucket the numerical measurement into intervals, and then to display the frequency (or...

Fragmented

2009-04-23T01:58:00.001-07:00

In the process of planning the syllabus for my next PhD course on "Scientific Data-Collection", to be offered for the third time in Spring 2009, I have realized how fragmented the education of...

Collecting online data (for research)

2009-04-17T18:51:00.001-07:00

In the new era of large amounts of publicly available data, an issue that is sometimes overlooked is ethical data collection. Whereas for experimental studies involving humans we have clear...

Principal Components Analysis vs. Factor Analysis

2009-03-25T23:27:00.000-07:00

Here is an interesting example of how similar mechanics lead to two very different statistical tools. Principal Components Analysis (PCA) is a powerful method for data compression, in the sense of...

Are experiments always better?

2009-03-24T23:23:00.000-07:00

This continues my "To Explain or To Predict?" argument (in brief: statistical models aimed at causal explanation will not necessarily be good predictors). And now, I move to a very early stage in the...

What R-squared is (and is not)

2009-03-10T08:01:00.000-07:00

R-squared (aka "coefficient of determination", or for short, R2) is a popular measure used in linear regression to assess the strength of the linear relationship between the inputs and the output. In...

Start the Revolution

2009-03-09T07:00:00.000-07:00

Variability is a key concept in statistics. The Greek letter Sigma has such importance, that it is probably associated more closely with statistics than with Greek. Yet, if you have a chance to...

Beer and ... crime

2009-01-09T13:53:00.000-08:00

I often glimpse the local newspapers while visiting a foreign country (as long as it is in a language I can read). Yesterday, the Australian Herald Sun had the article "Drop in light beer sales...

Microsoft and the financial downfall

2008-10-17T20:08:00.000-07:00

One of the misleading features of Microsoft Office software is that it gives the user the illusion that they are in control of what's visible and what's hidden to readers of the files. One example is...

Sensitivity, specificity, false positive and false negative rates

2008-10-07T08:07:00.000-07:00

I recently had an interesting discussion with a few colleagues in Korea regarding the definition of false positive and false negative rates and their relation to sensitivity and specificity....

Dr. Doom and data mining

2008-09-21T21:08:00.000-07:00

Last month The New York Times featured an article about Dr. Doom: Economics professor "Roubini, a respected but formerly obscure academic, has become a major figure in the public debate about the...

Data conversion and open-source software

2008-09-03T08:17:00.000-07:00

Recently I was trying to open a data file that was created in the statistical software SPSS. SPSS is widely used in the social sciences (a competitor to SAS), and appears to have some ground here in...