Bzst...

The value of p-values: Science magazine asks

2009-11-06T19:09:00.001-08:00

My students know how I cringe when I am forced to teach them p-values. I have always felt that their meaning is hard to grasp, and hence they are mostly abused when used by non-statisticians. This is...

Testing directional hypotheses: p-values can bite

2009-10-26T19:32:00.001-07:00

I've recently had interesting discussions with colleagues in Information Systems regarding testing directional hypotheses. Following their request, I'm posting about this apparently illusive issue....

SAS On Demand: Enterprise Miner -- Update

2009-10-08T13:49:00.001-07:00

Following up on my previous posting about using SAS Enterprise Minder via the On Demand platform: From continued communication with experts at SAS, it turns out that with the EM version 5.3, which is...

SAS On Demand: Enterprise Miner

2009-10-02T15:48:00.001-07:00

I am in the process of trying out SAS Enterprise Miner via the (relatively new) SAS On Demand for Academics. In our MBA data mining course at Smith, we introduce SAS EM. In the early days, we'd get...

Interpreting log-transformed variables in linear regression

2009-09-14T18:28:00.001-07:00

Statisticians love variable transformations. log-em, square-em, square-root-em, or even use the all-encompassing Box-Cox transformation, and voilla: you get variables that are "better behaved". Good...

Creating color-coded scatterplots in Excel: a nightmare

2009-08-30T20:04:00.001-07:00

Scatterplots are extremely popular and useful graphical displays for examining the relationship between two numeric variables. They get even better when we add the use of color/hue and shape to...

Data Exploration Celebration: The ENBIS 2009 Challenge

2009-08-20T10:23:00.001-07:00

The European Network for Business and Industrial Statistics (ENBIS) has released the 2009 ENBIS Challenge. The challenge this time is to use an exploratory data analysis (EDA) tool to answer a bunch...

Histograms in Excel

2009-06-13T04:40:00.001-07:00

Histograms are very useful charts for displaying the distribution of a numerical measurement. The idea is to bucket the numerical measurement into intervals, and then to display the frequency (or...

Fragmented

2009-04-23T01:58:00.001-07:00

In the process of planning the syllabus for my next PhD course on "Scientific Data-Collection", to be offered for the third time in Spring 2009, I have realized how fragmented the education of...

Collecting online data (for research)

2009-04-17T18:51:00.001-07:00

In the new era of large amounts of publicly available data, an issue that is sometimes overlooked is ethical data collection. Whereas for experimental studies involving humans we have clear...

Principal Components Analysis vs. Factor Analysis

2009-03-25T23:27:00.000-07:00

Here is an interesting example of how similar mechanics lead to two very different statistical tools. Principal Components Analysis (PCA) is a powerful method for data compression, in the sense of...

Are experiments always better?

2009-03-24T23:23:00.000-07:00

This continues my "To Explain or To Predict?" argument (in brief: statistical models aimed at causal explanation will not necessarily be good predictors). And now, I move to a very early stage in the...

What R-squared is (and is not)

2009-03-10T08:01:00.000-07:00

R-squared (aka "coefficient of determination", or for short, R2) is a popular measure used in linear regression to assess the strength of the linear relationship between the inputs and the output. In...

Start the Revolution

2009-03-09T07:00:00.000-07:00

Variability is a key concept in statistics. The Greek letter Sigma has such importance, that it is probably associated more closely with statistics than with Greek. Yet, if you have a chance to...

Beer and ... crime

2009-01-09T13:53:00.000-08:00

I often glimpse the local newspapers while visiting a foreign country (as long as it is in a language I can read). Yesterday, the Australian Herald Sun had the article "Drop in light beer sales...

Microsoft and the financial downfall

2008-10-17T20:08:00.000-07:00

One of the misleading features of Microsoft Office software is that it gives the user the illusion that they are in control of what's visible and what's hidden to readers of the files. One example is...

Sensitivity, specificity, false positive and false negative rates

2008-10-07T08:07:00.000-07:00

I recently had an interesting discussion with a few colleagues in Korea regarding the definition of false positive and false negative rates and their relation to sensitivity and specificity....

Dr. Doom and data mining

2008-09-21T21:08:00.000-07:00

Last month The New York Times featured an article about Dr. Doom: Economics professor "Roubini, a respected but formerly obscure academic, has become a major figure in the public debate about the...

Data conversion and open-source software

2008-09-03T08:17:00.000-07:00

Recently I was trying to open a data file that was created in the statistical software SPSS. SPSS is widely used in the social sciences (a competitor to SAS), and appears to have some ground here in...

Simpson's Paradox in Bhutan

2008-08-25T00:13:00.000-07:00

This year I am on academic sabbatical, hence the lower rate of postings. Moreover, postings this year might have an interesting twist, since I am in Bhutan volunteering at an IT Institute. As part of...

Summaries or graphs?

2008-06-10T16:44:00.000-07:00

Herb Edelstein from Two Crows consulting introduced me to this neat example showing how graphs are much more revealing than summary statistics. This is an age-old example by Anscombe (1973). I will...

Resources for instructors of data mining courses in b-schools

2008-06-10T07:26:00.000-07:00

With the increasing popularity of data mining courses being offered in business schools (at the MBA and undergraduate levels), a growing number of faculty are becoming involved. Instructors come from...

Student network launched!

2008-06-06T09:03:00.000-07:00

I recently launched a forum for the growing population of MBA students who are veterans of our data mining course. The goal of the forum is to foster networking, job-related communications (not too...

Weighted nearest-neighbors

2008-05-31T19:36:00.000-07:00

K-nearest neighbors (k-NN) is a simple yet often powerful classification / prediction method. The basic idea, for predicting a new observation, is to find the k most similar observations in terms of...

Why zipcodes take over trees

2008-05-13T09:20:00.000-07:00

I few weeks ago I went up to West Point to present a talk at their 2008 Statistics Workshop. Another speaker was Professor Wei-Yin Loh, from Univ of Wisconsin. He gave a very interesting talk that...