Psychological Statistics

The aesthetics of error bars

2012-05-27T21:28:00.000+01:00

This blog and my other main blog (the companion blog for my book) are now syndicated via R-bloggers (posts tagged R only) and statsblogs.com. The latter is a relatively new blog aggregator but looks to have some interesting content. R-bloggers it quite well established and I was already an occasional reader.

Looking at some recent content I noticed an interesting piece by Ben Bolker (author, among other things, of the excellent bbmle package in R) on dynamite plots. Until a few years ago (possibly in the early research for my book) I had not heard the term 'dynamite plot' or the negative press the attract in some research fields. In my own discipline (psychology) and in experimental psychology in particular bar plots with error bars (looking like sticks of dynamite stacked in a row) are rather popular. In fact, I was taught to use them in preference to dot plots when plotting interactions in ANOVA (their main application in experimental psychology). The main arguments against dot plots are that it is easy to manipulate them to make effects look large large by adjusting the scale (and sometimes software does this automatically). The advantage of switching to a bar plot is that these are supposed to be zero-referenced and thus effects are likely to more appropriately scaled.

Here is an example of a dynamite plot adapted from chapter 3 of my book:

(Colleagues of mine will note that the quantity displayed by the error bars is not labeled. This should always be clear from plot or figure caption - here they are 95% CIs).

Some of the material on dynamite plots on the web is somewhat one-sided (e.g., see here, here, here or the comments here). Ben Bolker bravely presents a more balanced picture. He also gets to the heart of the issue by noting that most criticisms of dynamite plots suggest box plots or plots of raw data as alternatives. This doesn't seem appropriate if your goal is inference rather than description. As Bolker notes, if you've decided to something like ANOVA you are already implicitly assuming approximate normality of the errors and so forth. Thus if the main purpose of the plot is inferential or to display key patterns among the data, box plots or raw data plots are not so useful. (Don't get me wrong I think think that plotting raw data is a good idea - but exploratory work and model checking are different from inference). So for a plot of means with error bars, the choice of dot plot or bar plot is one of aesthetics. These days my preference is for dot plots (which are more versatile and have a better information to ink ratio), but I think a well constructed dynamite plot can be appropriate in some situations. I would usually save these for a situation in which the pattern was quite simple (e.g., a 2 by 2 interaction), there was a meaningful zero or other reference point and when my audience are familiar with this style of plot and may prefer them.

A further aesthetic point here is how to plot the error bars themselves. I am persuaded by Andrew Gelman's argument that the crossbars on conventional error bar plots are ugly and counterproductive. They draw your attention to the extremes of the error bar - when values closer to the statistic being estimated are more plausible. Here is the earlier dyamite plot redrawn as a conventional error bar plot and in cleaner Gelman-approved style:

I find the version on the right to be much prettier. Furthermore it makes it easier to adapt them into two-tiered error bar plots. I like to use two-tier plots to convey 95% CIs for individual means (outer tier) and inferential (difference-adjusted) 95% CIs (inner tier). The inner tier approximates to a 95% CI for the difference - so that the means can be considered different by conventional criteria if the inner tier error bars don't overlap:

In my paper on within-subject CIs I used the style on the left. However, with hindsight I wish I'd included the style on the right. Varying the width of the bars avoids the ugly crossbars but may make detecting a 'statistically significant' difference trickier. I think that aesthetics win here because graphical methods aim to support informal inference - they are not supposed to be there for fine-grain, formal inference (which can be supported by formal hypothesis tests of various kinds - not just null hypothesis significance tests).

Serious stats - free statistics resources

2012-03-23T20:52:00.000Z

The companion web site for Serious Stats is now live:

http://www.palgrave.com/psychology/baguley/

The web site includes:

- a free sample chapter (Chapter 15: Contrasts)
- data sets
- R scripts
- 5 online supplements (for meta-analysis, multiple imputation, replication probabilities, pseudo-R squared and loglinear models)

Also don't forget the Serious stats blog to accompany the book.

Graphing between-subject confidence intervals for ANOVA

2012-03-19T11:58:00.002Z

This is a quick follow up to my earlier post that discussed how to graph CIs for within-subjects (repeated measures) ANOVA designs. My forthcoming book Serious stats describes how to do this for between-subjects designs (a much simpler problem). The blog that accompanies the book now has a post summarizing the main options and explaining how to plot difference-adjusted CIs (95% CIs constructed so that non-overlapping intervals correspond to a statistically significant difference between means at p < .05). In addition, the post includes R functions to calculate and plot difference-adjusted CIs (though the calculations are not difficult to reproduce by hand).

p curves revisited

2012-03-15T23:35:00.000Z

I finally found some time to take a closer look at p curves. I haven't had a chance to follow-up my simulations (and probably won't for a few weeks if not months), but I have had time to think through the ideas the p curve approach raises based on some of the comments I've received and a brief exchange with Uri Simonsohn (who has answered a few of my questions).

First, I got a couple of things at least partly wrong.

i) how p curves work

ii) the potential for correlated p values

How p curves work

I made the (I think) reasonable assumption that p curve analysis involved focusing on a bump just under the p = .05 threshold. Other work (Wicherts et al., 2011) has shown that there is indeed some distortion around this value. My crude simulation suggested that p curves could maybe be used to detect this kind of bump - but that the method was noisy and required large N.

All good so far except my assumption was completely wrong. This isn't what Simonsohn and colleagues are proposing at all. They are focusing on the whole of the distribution between p = 0 and p = .05. This is a very different kind of analysis because it uses all the available p value information about 'p hacking' (if you accept the highly plausible premise that p hacking is concentrated on statistically significant p values).

Null effects will therefore produce a flat p curve (because the distribution of p under the null is uniform). Simonsohn argues that non-null effects should produce downward sloping p curves. He and his colleagues have simulated p curves under various ranges of effect size to confirm this - and there is also an analytic proof for the normal case (Hung et al., 1997).* I also (inadvertently) confirmed this in my original simulations - which show the downward sloping trend (but note that I include p values up to p = .10 in my plots).

However, mixing in p hacked studies to a flat curve will produce an upward sloping curve - the feature that Simonshohn and his colleagues are focusing on. I haven't simulated this directly - but it seems sensible because p hacking is (in essence) a flavour of optional stopping (adding data or iterating analyses until you squeeze a statistically significant effect out). Certainly, an upward sloping curve would be a signal of something wierd going on.

This approach uses more information than my mistaken 'p bump' approach and so should be much more stable.

* It is far from unreasonable to treat the distribution of effects as approximately normal - as is common in meta-analysis (and see also Gillett, 1994), but I don't think the pattern depends strongly on this assumption.

Correlated p values

It is well known that p values are inherently extremely noisy 'statistics' - they jump around all over the place for identical replications. Geoff Cumming and colleagues have published some good work on this (e.g., Cumming & Fidler, 2009). Thus the same effect in different studies or different effects of similar sizes will in general not tend to have correlated p values. However, the noise that causes this jumping around will be crystalized if you use the same data to re-calculate the p value. This could cause correlated p values where data is re-used or where variables are very highly correlated. For example, this could happen if you add a covariate that is a modest predictor of Y and uncorrelated with and report p values with and without the covariate. It could also happen if you report essentially the same analysis twice with a very similar variable (e.g., X correlated with children's age or X correlated with years of schooling).

There are two main solutions here: a) just filter out p values that re-use data or use highly-correlated data, or b) model the correlations in some way by accounting for within-study clustering - as you might in a multilevel model and some forms of meta-analysis (itself a form of multilevel model).

In summary, I think the p curve approach looks very interesting, and I'd certainly like to see more work on it (and hope to see the full version published some time soon).

References

Cumming, G., & Fidler, F. (2009). Confidence Intervals. Zeitschrift für Psychologie / Journal of Psychology, 217(1), 15-26.

Gillett, R. (1994). Post hoc power analysis. Journal of Applied Psychology, 79(5), 783-785. doi:10.1037//0021-9010.79.5.783

Hung, H. M., O’Neill, R. T., Bauer, P., & Köhne, K. (1997). The behavior of the P-value when the alternative hypothesis is true. Biometrics, 53(1), 11-22.

Wicherts, J. M., Bakker, M., & Molenaar, D. (2011). Willingness to share research data is related to the strength of the evidence and the quality of reporting of statistical results. PloS one, 6(11), e26828. doi:10.1371/journal.pone.0026828

R code for p curves

2012-03-14T20:23:00.000Z

I have finally got around to posting the R code for my p curve simulation. Those familiar with R will realize how crude it is (I've been caught up with other urgent stuff and had no time to explore further).

You are welcome to play with (and improve!) the code. Changing delta will alter the (at present) fixed effect size. It would be more realistic to vary this (and the sample sizes). A good starting point for the effect size distribution (in the population) might be a normal distribution with say a mean of zero and a variance of 1 (see Gillett, 1994).

delta <- 0.5
m1 <- 10
sd <- 2
m2 <- m1 + sd*delta
n1 <- n2 <-25
 
n.sims <- 500
p.data <- replicate(n.sims, t.test(rnorm(n1, m1,sd), rnorm(n2, m2,sd))$p.val, simplify=T)
 
par(mfrow=c(5,3))
for (i in 1:15) {
 p.data <- replicate(n.sims, t.test(rnorm(n1, m1,sd), rnorm(n2, m2,sd))$p.val, simplify=T)
 hist(p.data, xlim=c(0,0.1), breaks = 99, col = 'gray')
 }

R code html script courtesy of Pretty R at inside-R.org

References

Gillett, R. (1994). Post Hoc Power Analysis. Journal of Applied Psychology, 79, 783-785.

Simulating p curves and detecting dodgy stats

2012-02-16T21:08:00.000Z

Psych your mind has an interesting blog post on using p curves to detect dodgy stats in a a volume of published work (e.g., for a researcher or journal). The idea apparently comes from Uri Simonsohn (one of the authors of a recent paper on dodgy stats). The author (Michael W. Kraus) bravely plotted and published his own p curve - which looks reasonably 'healthy'. However, he makes an interesting point - which is that we don't know how useful these curves are in practice - which depends among other things on the variability inherent in the profile of p values.

I quickly threw together a simulation to address this in R. It is pretty limited (as I don't have much time right now), but potentially interesting. It simulates independent t test p values where the samples are drawn from independent, normal distributions with equal variances but different means (and n = 25 per group). The population standardized effect size is fixed at d = 0.5 (as psychology research generally reports median effect sizes around this value). Fixing the parameters is unrealistic, but is perhaps OK for a quick simulation.

I ran this several times and plotted p curves (really just histograms with bins collecting p values at relevant intervals). First I plotted for an early career researcher with just a few publications reporting 50 p values. I then repeated for more experienced researchers with n = 100 or n = 500 published p values.

Here are the 15 random plots for 50 p values:

At least one of the plots has a suspicious spike between p = .04 and .05 (exactly where dodgy practices would tend to push the p values).

What about 100 p values?

Here the plots are still variable (but closer to the theoretical ideal plotted on Kraus' blog).

You can see this pattern even more clearly with 500 p values:

Some quick conclusions ... The method is too unreliable for use with early career researchers. You need a few hundred p values to be pretty confidence of a nice flat pattern between p = .01 and p = .06. Varying the effect size and other parameters might well inject further noise (as would adding in null effects which have a uniform distribution of p values and are thus probably rather noisy).

I'm also skeptical that this is useful for detecting fraud (as presumably deliberate fraud will tend to go for 'impressive' p values such as p < .0001). Also (going forward) fraudsters will be able to generate results to circumvent tools such as p curves (if they are known to be in use).

On nonparametric statistics ...

2012-02-14T23:54:00.003Z

I'm not a big fan of the term "nonparametric statistics", or at least how it is used in psychology and related fields (e.g., education and health research). This is one reason why I don't make a big deal of the parametric/non-parametric distinction in my Serious stats book and probably partly why a recent article in APS Observer annoyed me so much.

Why did it annoy me so much? It says, in essence, what many standard psychology text books say (and also makes several good points - which I shall largely ignore for rhetorical purposes). I like in particular the points about statistical tests used to check statistical assumptions lacking power and making statistical assumptions themselves (which are typically unchecked and they are not robust to). Indeed, that's another thing that annoys me and I have blogged/bored people about in the past (e.g., see here). I also think the article might have been chopped about a bit by editors (as it contains at least one important reference not cited and some of my gripes are also contained in references cited by the article I'm annoyed with). So, on balance, it is (by psychology mainstream standards) not at all a bad piece. However, by buying into the standard nonparametric stats presentation it perpetuates a few myths or errors and may inadvertently gives some pretty poor advice mixed in with the good.

Here is a quick list of my criticisms:

(1) The definition of nonparametric statistics is deeply confused. The author starts by writing: "Nonparametric statistical analyses are used to investigate research questions in which the dependent variable is ranked or categorical rather than quantified in a true numeric sense". This seems to suggest that non-parametric procedures are defined by having a discrete or bounded DV. Later it adds that "Traditional parametric statistics require a number of assumptions about the characteristics (i.e., parameters) of the data." This is an appeal to the idea that parametric statistics assume a particular probability distribution (the parameters of which are estimated by the data). This seems like a better definition to me, although like many psychologists, the author appears to assume that the probabilty distribution assumed is always a normal distribution. Mixing the two aspects of the definition is confusing. It is easy to find statistical procedures that are parametric in the second sense, but involve ranked or categorical DVs. I would argue that a chi-square test of independence or a sign test is parametric by the second definition. Not only is the definition problematic, but it could lead to poor analytic decisions. For instance, dichotomous outcomes are often best analyzed using parametric techniques such as logistic regression (a generalized linear model with a logistic link function and a binomial random component) rather than the methods surveyed in the article.

(2) The article also reads as if nonparametric tests are the same thing as rank randomization tests. Rank randomization tests are examples of nonparametric tests in the sense they are distribution free (making no assumption about a particular probability distribution for the data). However, there are many other nonparametric methods that don't involve ranks. Rank randomization tests are useful but limited in scope. In addition, if the raw data are in the form of ranks then a rank transformation is pretty pointless (you might as well jut run a regular test).

The main limitation of the rank approaches is that the rank transformation is irreversible; it destroys the link between the analysis and the raw scores. This is a serious problem if you care about the raw scores - perhaps because you want to test whether effects are non-additive or because you want to get an interval estimate or a effect sizes on the original scale. There are ways to help you do this with rank transformation procedures, but they are generally pretty fiddly. There are good reasons why people don't tend to use rank transformations in multiple regression, use them to test interaction effects or use them to construct confidence intervals.

Third, rank randomization (and other nonparametric) procedures are bound by pesky assumptions! They tend to make weaker assumptions than parametric procedures, but all statistical procedures make assumptions. The assertion that the "main assumptions of nonparametric tests are that the dependent variable should be continuous and have independent random sampling, which means that nonparametric statistics do not require assumptions of homogeneity of variance and normality" is misleading. The precise assumptions vary from test to test and with the hypothesis being tested. As a rule, rank randomization and related rank transformation tests assume that samples with similar shapes of distribution are being compared. They can therefore be undermined by heterogeneity of variance or varying degrees of skew. If the distributions have very dissimilar shapes then the tests can sometimes behave very strangely (e.g., it is possible to get outcomes where A > B, B > C and yet C > A). Continuity is also not necessarily a requirement of the DV or of the underlying construct being measured (though it may be for some hypotheses).

(4) Many generalizations are made that simply don't hold up: "When the data violates the assumptions of a parametric test, nonparametric tests are again the more powerful analytic technique (Siegel, 1957)." This doesn't follow. Often this is true, but not always. There has been a lot of research on this topic since 1957 and parametric tests are not always inferior. This is particularly true if you don't restrict parametric techniques to t tests, linear regression and ANOVA. Some nonparametric techniques are known to have very poor power. I also got irritated by the critique of log and similar transformations that states: "while these transformations can make variables more normally distributed, they can also diminish or alter experimental effects, which can reduce power." To the extent this is true, it is also true of the rank transformation (more so in some cases). Furthermore, the real problem is with arbitrary transformations. Log transformations - where appropriate - tend to aid interpretation of effects (e.g., by quantifying them as proportionate rather than additive effects). I also dislike the implication that "experimental effects" exist in a pristine form prior to transformation. This is simply not the case - how to quantify and scale measurement of an effect is a tricky business (e.g., memory researchers can use percentage correct, hits minus false alarms, A prime d prime etc.) and many effects come "pre-transformed" (e.g., measurement of loudness in decibels). Transformations (including the rank transformation) are useful tools that can increase power and aid interpretation if used carefully and appropriately.

(5) The use of parametric statistics to analyze Likert-style rating scales may be one of the "seven deadly sins of statistical analysis", but it is rarely a big problem in practice. Least squares methods are most messed up by heavy tailed distributions, severe skew or outliers. If anything, Likert-style rating scales tend not to have these problems (or to manifest them relatively mildly). Furthermore, where there are problems with Likert-style measures, rank randomization or transformation tests s are probably not the solution. A number of parametric procedures for ordinal outcomes exist - notably ordered logistic regression (though least squares methods such as t tests, ANOVA or regression should work well when their assumptions are not badly violated).

(6) The variety of nonparametric tests referred to is slightly artificial. As a general rule there are advantages to sticking to standard 'parametric' tests such as the Welch-Satterthwaite t test or one-way ANOVA rather than using named rank-transformation tests such as the Mann-Whitney U test. In some cases there may be advantages with specialized rank randomization tests where sample sizes are small (e.g., because software such as R implements exact versions). However, there are a few cases where the rank randomization tests are not robust (e.g., Mann-Whitney U test is not robust to heterogeneity of variance) or lack power (e.g., the Friedman test, Page's L test and most multiple comparison procedures available for ranks). Rank transforming the data and then running a t test with Welch-Satterthwaite correction is superior to running the Mann-Whitney directly (Zimmerman & Zumbo, 1993a). For more on the low power of the Friedman test and better alternatives see here.
Rather than think in terms of nonparametric statistics, it is better to focus on checking assumptions (using graphical methods and simple descriptives) and checking our models against more robust procedures. If more robust methods show different results - the next step is to find out why (and definitely not report just the outcome you prefer). This should lead you to a superior model (using robust methods or perhaps a more appropriate parametric model). The consideration of robust methods is particularly important. This includes some rank transformation tests, but also includes robust regression, bootstrapping and other tools (e.g., see Wilcox & Keselman, 2003; 2004; Baguley, 2012).

References

Baguley, T. (2012, in press). Serious stats: A guide to advanced statistics for the behavioral sciences. Basingstoke: Palgrave.

Wilcox, R. R., & Keselman, H. J. (2003). Modern robust data analysis methods: Measures of central tendency. Psychological Methods, 8, 254-274.

Wilcox, R. R.,& Keselman, H. J. (2004). Robust regression methods: Achieving small standard errors when there is heteroscedasticity. Understanding Statistics, 3, 349- 364.

Zimmerman, D. W., & Zumbo, B. D. (1993b). Rank transformations and the power of the Student t test and Welch t' test for non-normal populations with unequal variances. Canadian Journal of Experimental Psychology, 47, 523-539.

Comparing correlations update

2012-02-05T17:42:00.002Z

I have just published R code for calculating CIs for differences between correlations on the Serious stats book blog. This covers independent correlations (taken from chapter 6 of the book) and dependent correlations (new R code written as a supplement to chapter 6).

UPDATE on the update ...

I have also added an Excel spreadsheet that should match the R output (though the latter is probably more accurate and reliable).

Serious Stats book and blog update

2012-02-02T16:50:00.001Z

This is a quick update to announce my new blog Serious Stats. This is a companion to my forthcoming book of the same name:

Baguley, T. (2012, in press). Serious stats: A guide to advanced statistics for the behavioral sciences. Basingstoke: Palgrave.

I think is better to separate book specific content out from my regular posts (though in some cases this will be a bit fuzzy). I will also try and post short updates here when something relevant gets published on the blog for the book.

More on "A problem of significance"

2012-02-01T20:38:00.009Z

A longer version of my earlier post A problem of significance just appeared in The Psychologist.

Baguley, T. (2012). Can we be confident in our statistics? The Psychologist. 25, 128-9.

Shortly after publication I received an email asking about statistical analysis of differences in correlations. This is more tricky than you might think. I'm working on some R code to implement one of the better approaches and plan to blog on this shortly ...

(See update here.)

Calculating and graphing within-subject confidence intervals for ANOVA

2011-10-04T15:43:00.006+01:00

Psychologists are gradually coming round to the view that it is a good idea to present interval estimates alongside point estimates of statistics. The most common statistic reported in psychology research is almost certainly the mean (strictly the arithmetic mean). Presenting an interval estimate for the mean of a single sample is usually quite simple. This is usually done as 95% confidence interval about the mean – and most researchers in psychology are able to calculate this by hand or get their statistical software to calculate and graph it for them.

Extending this to more than one mean introduces an additional layer of complexity. This is because the difference between two means is a different quantity, and its CI (although related to those of the individual means) is different in width from the CIs of the individual means. This creates a problem when plotting the CI because a researcher might be interested in the CI for an individual mean, the CI for their difference (or both).

The complexity increases further if the aim is to plot a set of means (e.g., from an ANOVA design). In this case, plotting all the possible differences (as is commonly done) obscures patterns in the individual means (e.g., linear or quadratic trends). Last, but not least, if the means are not from independent samples, there are further difficulties. This happens in within-subjects or repeated measures designs.

In these designs the variation around each mean is correlated with the variation around the other means. This correlation arises from individual differences. Statistical procedures such as ANOVA can capitalize on these individual differences to produce more sensitive statistical inferences (i.e., to increase statistical power or obtain narrower CIs). This is done by estimating the variation due to individual differences, and removing it from the error variance (the estimate of statistical noise in the data set).

This is a problem for graphical presentation of means because the precision of individual means is influenced by individual differences, whereas the precision of differences between means is not (because the estimate of individual differences is common to repeated samples from the same people and thus can be removed). Further complications arise when the sphericity assumption of repeated measures ANOVA is violated.

Several solutions to these problems have been proposed in the literature. The best known of these in psychology is that of Loftus and Masson (1994). Another well-known solution is that of Goldstein and Healy (1995), extended to correlated samples by Afshartous and Preston (2010).

Despite a large literature on the problems of graphing a set of correlated means, many people avoid the problems altogether by not reporting (or graphing) CIs or report CIs that are misleading in some way. Researchers are often unaware of the problems or find the solutions hard to understand and implement.

I recently reviewed the main approaches in the literature, describe how to obtain suitable intervals for individual means and differences between means and provide R code to calculate and plot the intervals.

The main highlights are that:

i) for inferences about individual means the standard approach works fairly well for between-subject (independent measures) designs, but there is a case to use CIs from a multilevel model for within-subject (repeated measures) designs

ii) an approach proposed by Cousineau (2005) with a correction by Morey (2008) offers advantages over the Loftus and Masson (1994) approach for within-subject ANOVA designs. It simplifies the calculations and does not assume sphericity. The Loftus-Masson approach will however usually be superior when n is small.

iii) if you are interested in differences between means then you should probably plot a version of the Cousineau-Morey (or Loftus-Masson) interval that is adjusted so that overlap of the CIs around two individual means corresponds to overlap of the CI for their difference. This can be done by incorporating a multiplier to the width of the individual CIs. This multiplier is equal to (2^0.5)/2.

iv) if you are interested in both precision of individual means and their differences you can use a two-tiered error bar to display both quantities (Cleveland, 1985).

v) the intervals (and graphical presentation of means) are useful for informal inference about a set of means. For formal inference it is better to set up precise hypotheses and test these via an a priori of contrast. This could be a traditional null hypothesis significance test, but other approaches are available. These include confidence intervals, Bayes factors, likelihood ratios and so forth (Baguley, in press; Dienes, 2008).

The paper is available here, the R code here and the data sets here.

Update: R functions now available for the simpler between-subjects (independent measures) ANOVA
case (at the Serious stats blog).

References

Afshartous D., & Preston R. A. (2010). Confidence intervals for dependent data: equating nonoverlap with statistical significance. Computational Statistics and Data

Analysis. 54, 2296-2305.

Baguley, T. (2011, in press). Calculating and graphing within-subject confidence intervals for ANOVA. Behavior Research Methods. DOI: 10.3758/s13428-011-0123-7

Baguley, T. (2012, in press). Serious Stats: A guide to advanced statistics for the behavioral sciences. Basingstoke: Palgrave.

Cleveland. W. S. (1985). Elements Of Graphing Data. New York, NY: Chapman & Hall.

Cousineau, D. (2005). Confidence intervals in within-subject designs: A simpler solution to Loftus and Masson’s method. Tutorials in Quantitative Methods for Psychology, 1, 42-45.

Dienes, Z. (2008). Understanding Psychology as a Science: An Introduction to Scientific and Statistical Inference. Basingstoke: Palgrave Macmillan.

Goldstein, H., & Healy, M. J. R. (1995). The graphical presentation of a collection of means. Journal of the Royal Statistical Society. Series A (Statistics in Society), 158, 175-177.

Loftus, G. R., & Masson, M. E. J. (1994). Using confidence intervals in within-subject designs. Psychonomic Bulletin & Review, 1, 476-490.

Morey, R. D. (2008). Confidence intervals from normalized data: A correction to Cousineau (2005). Tutorials in Quantitative Methods for Psychology, 4, 61-64.

A problem of significance

2011-09-15T23:27:00.006+01:00

Several people have drawn my attention to a recent article on a common error in published statistical analyses in neuroscience. Sander Nieuwenhuis, Birte Forstmann and Eric-Jan Wagenmakers published (in Nature Neuroscience) a critique of statistical analyses in the neuroscience literature. This paper has been written about by Ben Goldacre and Andrew Gelman (who published an article on the general problem some time ago) - so I won't go into too much detail.

The point of interest for me is that the error concerns something that most psychologists should know all about (and hence should be expected not to make the error). It concerns the case of two differences, one statistically significant and one non-significant. For example, group 1 may show a significant difference between experimental condition and placebo (for a drug intervention), while group 2 do not. A naive interpretation is that the drug works for group 1 but not group 2. This is not necessarily true. The proper test of a difference in effects of the drug between groups is an interaction test. Psychologists tend to avoid this error because we have heavily trained in ANOVA as undergraduates (certainly in the UK and probably also in the US and most of Europe). Even if we fail to learn this, reviewers and editors (in psychology) tend to spot the error.*

Are psychologists then entitled to feel a little bit smug? Perhaps, but only a little. First, I think the reason we are relatively good performers on this point is because we tend to view many statistical analyses through an "ANOVA" lens. Factorial ANOVA (in which factors are orthogonal) includes the interaction term by default. The 2 by 2 factorial ANOVA is the workhorse of experimental psychology. Our familiarity with this type of design and analysis makes this easy to spot. Second, our ANOVA lens leads to other errors - notably dichotomizing continuous variables (e.g., via median split) in order to squeeze them into an ANOVA design. This always decreases statistical power, and can - albeit infrequently - produce spuriously significant effects (see MacCallum et al., 2002). These errors are sometimes less serious than the difference of differences/interaction error (but are not harmless).

The real test then, is whether psychologists make the same (conceptual) error in a different context. The obvious context is that of association rather than difference. If males show a significant correlation between testosterone and aggression (e.g., r = .5, N = 25) and females don't (e.g., r = .3, N = 25), the correlation between testosterone and aggression is not significantly bigger for males than females. To confirm this you'd need to construct a test or (better still) confidence interval for the difference in correlations. This is hardly ever done - and, in my experience, psychologists frequently make this kind of claim without backing it up.** Methods for testing differences in correlations are a bit fiddly (e.g., depending on overlap or lack of overlap in the measurements), and rarely taught at undergraduate or even postgraduate level. The methods that are taught are also often a bit dodgy (see Zou, 2007, for some better alternatives).

Also note that (in both cases) the error can work the other way. Two correlations could be non-significantly different from zero but different from each other (e.g., r = .5 and r = -.5 with N = 10).

Postscript

There is, I think, a lesson or two here. A minor lesson is that interactions are bit more complicated than psychologists (particularly those very familiar with ANOVA) often think. I could write more on this (and do a bit in my forthcoming book). A major lesson is that this concept (the difference between significant and non-significant is not necessarily also statistically significant - see Gelman & Stern, 2006) is probably quite tricky. It may be worth exploring why ... I suspect it is because of several factors.

See updates here and here.

References

Gelman, A., & Stern, H. (2006). The difference between “significant” and “not significant” is not itself statistically significant. American Statistician, 60, 328–331.

MacCallum, R. C., Zhang, S., Preacher, K. J., & Rucker, D. D. (2002). On the practice of dichotomization of quantitative variables. Psychological Methods, 7, 19-40.

Nieuwenhuis, S., Forstmann, B. U., & Wagenmakers, E.-J. (2011). Erroneous analyses of interactions in neuroscience: A problem of significance. Nature Neuroscience, 14, 1105-1107.

Zou, G. Y. (2007). Toward using confidence intervals to compare correlations. Psychological Methods, 12, 399-413.

* Do I have any support for this position? Yes: anecodotal support (e.g., from editing or reviewing many dozens of papers) and some support from Nieuwenhuis et al. They found the error more prevalent in cellular and molecular neuroscience. ANOVA is core training in psychology and widely used in cognitive and behavioural neuroscience - and I'd argue that this reflects the influence of psychologists working in this area and of neuroscientists trained in and using similar methods.

** Do I have any support for this position? A little. It is easy to find basic psychology texts with ANOVA but without tests of differences in correlations being mentioned. It is rare to find tests of CIs of differences in published papers.

R: An introduction for psychologists

2011-01-13T13:50:00.000Z

Here are the slides from the Introduction to R session Danny Kaye and I ran at the BPS Mathematics, Statistics & Computing section CPS Workshop (13 December 2010, Nottingham Trent University).

Introduction to using R in research

2010-05-13T16:21:00.000+01:00

I was recently asked to give a talk to our graduate school annual conference. I offered several titles and the one they picked was Using R in research. I'm not sure if this was a good idea or not. The graduate school covers PhD students across three areas of the university: social sciences (including psychology), law and business. In addition the students tend to specialize in either qualitative or quantitive research methods, so I was talking to an audience who might know nothing about statistics or a great deal (e.g., several students have completed MSc courses in psychological research methods here or elsewhere).

My solution was try and explain the advantages of R relative to alternatives such as SPSS (probably the most common statistic package in the University). I also focussed a lot on graphical methods and simulation. It seemed to go quite well, but I worry that quite a few members of the audience were overwhelmed by large chunks of it.

I promised to put my slides on my blog - though I am not sure how useful they are to anybody who wasn't there. Without my commentary some (possibly most) of the slides won't make much sense. I spent a good deal of the time talking through exploratory plots of one data set (from Hayden, 2005). I use this example a lot in teaching and it involves a bit of class participating (guessing the origin of the data) - so I won't go into to detail here (lest I spoil it for future students), but you can google the original article if you are curious. I also spent some time on how R works (e.g., object types, assignment, basic modeling, plotting functions). My reasoning was that many of the audience have no familiarity with non-GUI interfaces in software and without explaining the basics of the interface they will not have the faintest clue how R works. For those with some familiarity (e.g., SPSS syntax) the examples were selected to show how powerful R can be for things like exploratory graphics.

Several students ask about resources for learning R. I mentioned some in earlier blog posts, but for psychologists Li and Baron's web resources are a good place to start. The other major resource is probably Quick R, but there are hundreds of other places to look online (depending on what stuff you need most).

Example of plotting a serial position curve in R

2010-03-03T20:14:00.018Z

A while ago I wrote a co-wrote chapter for an introductory psychology text book Essential Psychology: A Concise Introduction. This is a book edited and written by members of the department where I work. My contribution was the chapter on human memory (cunningly titled Memory).

I produced several plots for the chapter (some of which got cut due to severe space restrictions). One that stayed in was a serial position curve. For this plot I used data from Postman and Phillips (1965).

I feel particular proud of this plot because I was just beginning to use and learn R at the time (as opposed to dabbling) and because I had had a really hard time getting hold of the data. I first tried google, but had no joy (for some reason I thought someone would have put the raw data online, as it is a classic study - though maybe I just missed it). Then I searched for alternative data sets (as around that period there were quite a few similar studies). I was probably being too picky, but whatever the reason I had no luck.

It would have been trivial to make up fake data, but that didn't feel right. What I eventually did (and wished I'd done straight away) was print out the original figure and measure all the points by hand. I then entered these values into a spreadsheet and tweaked and remeasured until all the summary statistics matched those in the original paper to about one decimal place. This was a lot quicker than I had thought. I cheated slightly because I only needed data from the 20 word conditions (so I could leave out the 10 and 30 word conditions).

(I'm pretty sure I could have used computer software to capture the raw data from an image file, but I'd have had to find the software, learn how to use it and do all the checking anyway. For a single figure I'm reasonably sure measuring by hand would be faster.)

In re-plotting it I noticed a few things that I hadn't paid much attention to before. The main one was the authors report frequency of recalls for 18 participants with 6 lists each. This means all scores are out of 108 and I suspect lots of casual readers would (like me) assume they were percentages. For re-plotting I rescaled the data as percentages.

The plot itself just uses basic R functions. I'm writing about it because: i) I think it is a fairly clear illustration of how basic plot functions in R can produce what I think is a rather nice Figure. (The published version has been edited by the publisher, adding colour and making the style match figures in other chapters), ii) people may find it useful for teaching purposes. So please feel free to use and adapt the R code for non-commercial (e.g., teaching use).

First load the data from this .csv file (you will need to specify the path or change the working directory if the file is saved elsewhere).

pp65 <- read.csv("pp65.csv")

Then paste the following:

plot(pp65$SP, pch=NA, ylim=c(0,80), xlab= "Serial position", ylab= "Mean percentage recall", main = "Postman & Phillips (1965)", sub = '(20 word conditions only)')

points(pp65$C0, pch=19, col='black', cex=.7)

lines(pp65$C0, lty=3)

points(pp65$C15, pch=24, col='black', cex=.7)

lines(pp65$C15, lty=2)

points(pp65$C30, pch=22, col='black', cex=.7)

lines(pp65$C30, lty=5)

legend(3, 80, legend=c("No delay","15 second delay","30 second delay"), lty=c(3,2,5))

If you are new to R you can find out more about these plotting functions by using R help: ?par, ?plot, ?points and so on ...

References

Baguley, T., & Edmonds, A. J. (2010). Memory. In P. Banyard, M. N. O. Davies, C. Norman, & B. Winder (Eds.) Essential Psychology: A Concise Introduction (pp. 65-82). London: Sage.

Postman, L. & Philips, L. W. (1965). Short-term temporal changes in free recall. Quarterly Journal of Experimental Psychology, 17, 132-138.

Interaction plot from cell means

2010-02-25T00:23:00.001Z

I needed to produce a few a interaction plots for my book in R and, while the interaction.plot() function is useful it has a couple of drawbacks. First, the default output isn't very pretty. Second, it works from the raw data, whereas I often need plots from cell means. For teaching purposes it is quite common to produce plots without raw data (for hypothetical data or from published examples).

My first attempts at the plots involved setting them up element by element. Just going over some examples I decided to turn the basic plot (for a 2 x 2 ANOVA) into a simple function. Nothing fancy, just a regular interaction plot in black and white that I think is prettier than the SPSS, Excel or R defaults. At some point I may have a go turning it into a general I x J ANOVA plot (or maybe even add CIs, but I'll probably do that from raw data if I ever get round to it).

plot.2by2 <- function(A1B1,A1B2, A2B1, A2B2, group.names, legend = TRUE, leg.loc=NULL, factor.labels=c('Factor A', 'Factor B'), swap = FALSE, ylab= NULL, main = NULL){
group.means <- c(A1B1, A2B1, A1B2, A2B2)
if(missing(ylab)) ylab <- expression(italic(DV))
if(swap==TRUE) {
group.names <- list(group.names[[2]], group.names[[1]]) ; group.means <- c(A1B1, A1B2, A2B1, A2B2); factor.labels <- c(factor.labels[2], factor.labels[1])
}
plot(group.means, pch=NA, ylim=c(min(group.means)*.95, max(group.means)*1.025), xlim=c(0.8,2.2), ylab=ylab, xaxt='n', xlab=factor.labels[1], main=main)
points(group.means[1:2], pch = 21)
points(group.means[3:4], pch = 19)
axis(side = 1, at = c(1:2), labels = group.names[[1]])
lines(group.means[1:2], lwd = .6, lty = 2)
lines(group.means[3:4], lwd = .6)
if(missing(leg.loc)) leg.loc <- c(1,max(group.means))
if(legend ==TRUE) legend(leg.loc[1], leg.loc[2],legend = group.names[[2]], title = factor.labels[2], lty = c(3,1))
}

Call the function by entering the four cell means in conventional order: A1B1, A1B2 and so on where A1B1 is the mean of level 1 of factor A at level 1 of factor B. You also need a two item list containing text strings of the two level names of each factor. For instance:

lev.names <- list(c('A1', 'A2'), c('B1', 'B2'))
plot.2by2(5,15,10,20, lev.names)

You can swap the axes by adding the argument swap = TRUE:

plot.2by2(5,15,10,20, lev.names, swap = TRUE)

The default factor names are 'Factor A' and 'Factor B', but these are over-ridden in the call:

plot.2by2(5,15,10,20,lev.names, swap = TRUE, factor.labels= c('Factor 1','Factor 2'))

You can also change the y-axis label with ylab or add a main title with main. The legend can be dropped (legend = FALSE) if you don't want one or need it to be located outside the plot. To move the legend just specify coordinates with an argument such as leg.loc = c(1,10). You can also edit the source code directly. Here is an example with title and meaningful labels:

group.names <- list(c('placebo','drug'), c('male', 'female'))

plot.2by2(10,10,15,20, group.names, factor.labels=c('Drug', 'Sex'), swap = FALSE)

As this just uses basic plotting functions in R you can also manipulate the plot in other ways: adding lines with segments(), adding text with text() changing graphical parameters with par() and so on. Depending on your platform it is also easy to extract the plot as a .pdf or .jpg file. On a mac I save it as a .pdf file and open it in preview which allows me to save it as .png, .gif or whatever I need.

When correlations go bad ... (Or, I always wanted to write about the Society for the Suppression of the Correlation Coefficient)

2010-01-29T16:48:00.035Z

A while ago I promised a longer post on standardized effect size. This isn't it. Instead it is a link to my piece in the February 2010 issue of The Psychologist.

I had intended to write a short summary of my 2009 BJP paper, but that didn't really get off the ground. However, I had for a while wanted to write about the Society for the Suppression of the Correlation Coefficient. I had first read about this in Tukey's (1954) chapter on regression and path analysis. This is one of the earlier papers in the literature criticizing the preference for (standardized) correlation coefficients over (simple, unstandardized) regression coefficients. Read the article for an earlier example! I was reminded of its existence by Brillinger's (2001) paper.

As soon as I tried writing about the Society for the Suppression of the Correlation Coefficient things (I think) fell into place. The result is a piece that is one part history of statistics trivia, one part mini-tutorial and one part a summary of my 2009 paper.

For non-members of the BPS the link below contains a pre-publication version of 'When correlations go bad ...'.

I may also get around to the web site one day.

P.S. I adapted data and R code from Gelman and Hill (2007) for the example. I chose it because it a nice simple example of regression and because it is also a pointer to someone who argues in favour of standardization )at least in some situations). Both my 2009 paper and the 'When correlations go bad ...' are my attempts at getting people to rethink the use of correlation and standardization. The tone is deliberately (slightly?) polemic. There are other views so don't just take my word for it ... think about what you are trying to do and decide whether a correlation coefficient (or a standardized mean difference) is the right option.

References

Baguley, T. (2009). Standardized or simple effect size: What should be reported? British Journal of Psychology. 100, 603–617.

Baguley, T. (2010). When correlations go bad … The Psychologist, 23, 122-3.

Brillinger, D.R. (2001). John Tukey and the correlation coefficient, Computing Science and Statistics, 33, 204–218.

Gelman, A. & Hill, J. (2007). Data Analysis Using Regression and Multilevel/Hierarchical Models (Analytical Methods for Social Research). Cambridge: Cambridge University Press.

Tukey, J.W. (1954). Causation, regression and path analysis. In O. Kempthorne, T.A. Bancroft, J.W. Gowen & J.L. Lush (Eds.) Statistics and mathematics in biology (pp.35–66). Ames, IA: Iowa State College Press.

Strathclyde multilevel modeling talk

2010-01-25T09:40:00.003Z

I recently gave an introduction to multilevel modeling talk at Strathclyde University for PsyPAG. I promised to make my slides available. I'm still using powerpoint (and regretting it) so some of the symbols may be garbled. (I noticed that powerpoint turned all my tilda symbols to colons during the talk). So I hope it is readable:

Here is the link:

A statistical puzzle about averages II

2009-12-13T10:26:00.119Z

1. Who is correct? Professor B is correct. If the average family has 1.8 children then the average child would be expected to have more than 0.8 siblings.

2. Why? The average child is not from the average family. The concepts of average child and average family are different. For this reason there should no expectation that the average child should be from a family with an average number of children. Although there are restricted circumstances under which this can happen, they are sufficiently implausible to be discounted in any real world application (e.g., if all families have exactly the same number of children).

In one case the unit of analysis of the family and in the other it is the child. A concrete example may help. (I'll stick to defining average as arithmetic mean throughout, but the same logic extends to other averages such as the median - see the quotation from Kish below).

Assuming that the number of children varies between families (which it must do if the mean number of children per family is 1.8) then the average child will be from a family with a larger number of children than average. For example, imagine there are only four families:

		Number of siblings per child
Family	Number of Children	1^st child	2^nd child	3^rd child	4^th child
A	1	0	-	-	-
B	1	0	-	-	-
C	2	1	1	-	-
D	4	3	3	3	3

There are (1+1+2+4) = 8 children in the four families, thus the mean number of children per family is: 8/4 = 2.

It follows that the mean number of siblings per child is therefore:

(0+0+1+1+3+3+3+3)/8 = 14/8 = 1.75

So although each family has only two children (on average) each child has 1.75 siblings (not 1 sibling).

Note is that there are N = 4 families and n = 8 children. So switching the unit of analysis changes the denominator. Also note that while the families can plausibly be considered independent of each other the children can't (all children in the same family have the same number of siblings in this example, and more generally the number of siblings will be correlated).

What about zeroes? In the calculations above I excluded childless households as families. If you include only households without children as families the discrepancy would be larger.

Does it matter? Yes. Much real world data is clustered in this way. It is important to realize that the average is not going to be from the average (e.g., the average worker isn't from the average firm).

In practical terms this means that careful attention needs to be paid to sampling from families, workplaces, schools and so forth. A random sample of children will disproportionately sample children from large families. There are also social policy implications (e.g., if you are interested in reducing child poverty). Another example is that a random sample of schools will disproportionately sample small schools.

There are also other problems with analysis of clustered data. For this reason anyone working with clustered data needs to seriously consider using multilevel modeling or other methods that i) take into the clustering and ii) allow hypotheses about different levels of the model (e.g., children and families) to be explored.

Postscript

This is quite an old puzzle. I first came across this puzzle in the article by Jenkins and Tuten (1992). They include formulae for deriving one average from the other and cite Huntington (1924) and other later observations of the same phenomenon. I made the connection to multilevel models a little later. For a good (if slightly out of date) introduction to multilevel models see Snijders and Bosker (1999). Recently I noticed that Kish (1965) discusses the same phenomenon in passing.

This quote from Kish sets out the problem quite clearly:

Although the mean number of adults per household is only 2.02, the mean number of household members is 2.24 for the average adult. The greater size ranges of large organizations produce more striking effects. In 1960, 50 million people lived in 130 U. S. cities that had 100,000 or more population; in this population, the average city size was 0.39 million, but the size of the city in which the average person lived was 2.0 millions. Using medians does not help: the median city size was 0.19 million, but the median person lived in a city of 0.62 million.

Kish (1965, p. 571).

References

Jenkins, J. J., & Tuten, J. T. (1992). Why isn’t the average child from the average family? – and similar puzzles. American Journal of Psychology, 105, 517-526.

Kish, L. (1965). Sampling Organizations and Groups of Unequal Sizes, American Sociological Review, 20, 564-72.

Sniders, T. & Boskers, R. (1999). Multilevel Analysis: An Introduction to Basic and Advanced Multilevel Modeling. London: Sage.

A statistical puzzle about averages I

2009-12-09T22:46:00.002Z

I wrote this a few years ago for a departmental newsletter. For some reason the second part (with the answer) never got published. I stumbled across it almost by accident the other day and thought I'd share it. I'll publish the canonical answer in due course.

Professor Quack knows (from the UK census) that the average family has an arithmetic mean of 1.8 children. He also knows that (due to a bizarre mix-up in enrolment) that his Psychology For Everyone class is attended by a random sample of 50 people from the UK population. As part of an in-class demonstration of sampling theory he records the number of siblings of each student and calculates the average (using the standard formula for the arithmetic mean).

To his dismay he discovers that the students in his class have an arithmetic mean of 1.2 siblings. He later repeats the demonstration with two more random samples of the UK population (again with n = 50) and obtains values of 1.1 and 1.3 siblings.

Profesor Quack consults two of his colleagues: Professor A and Professor B.

Professor A replies thus:

“You are right to be dismayed. Bias has somehow entered either your sampling procedure or your calculation of the mean. The true mean number of siblings should be 1.8 - 1 = 0.8.”

Professor B interjects thus:

“Nonsense! All is as it should be. The expected number of siblings in a random sample is most certainly not 0.8. Rather, one would expect the average student to have more than 0.8 siblings, just as you have observed.”

1) Who is correct? Professor A or Professor B?

2) Why?

R functions for Dienes (2008) Understanding Psychology as a Science

2009-11-17T09:43:00.016Z

I recently wrote a review of Understanding psychology as a science: an introduction to scientific and statistical inference by Zoltan Dienes (2008). Dienes' book covers Neyman-Pearson null hypothesis significance testing, Bayesian inference and the likelihood method of inference (inspired by Fisher and associated with A. W. F. Edwards and more recently R. Royall).

One of the most useful features of the book is that Dienes provides Matlab code for examples of calculations in the book (e.g., for Bayes factors, likelihood intervals and so forth). This is not so useful for me because I don't use Matlab. Matlab licenses are also quite expensive and may not be possible for students to access it in many Psychology departments. For those without access to Matlab, Dienes also provides calculators for a number of functions on his own web page for the book. (The calculators are found by following the links to the appropriate chapter, so the Bayes factor calculator is found by following the Chapter 4 link).

Danny Kaye and I thought it would be useful to write R code to compliment the Matlab code for Dienes' functions as a 'bonus feature' for the review. As these functions and the notes for them take up quite a bit of space we decided to include only one, for a Bayes factor, in the review itself (with some notes on how to use it). Danny did most of the work writing functions, which are more-or-less direct translations of the original Matlab code (and have been checked against the web versions). The full set of functions is hosted on his web site along with the notes on how to use them. Also included are page references for the examples in the book.

Why did we write the R functions? First, they offer convenient access to the functions for teachers and students (because R is free and runs on Windows, Mac OS or Linux operating systems). Second, it reduces the burden on Dienes' web calculator (at a marginal decrease in ease of use). Third, R is open source so it is simple to see how the code works and to edit, extend or adapt it (though it is polite to acknowledge the authors of the original code). Fourth, we want to encourage more people to start using R!

As an example, I've already written some alternative functions for likelihood intervals (though as I happened I re-wrote these almost from scratch to get them to plot the likelihood function and interval and to take advantage of some built-in R functions). Those functions are intended for a the book I'm working on and so should appear in due course.

For those who are interested Danny and I are presently working on implementing Bayesian t tests in R (Bayes factors with objective priors) in a user-friendly way for researchers.

References:

Baguley, T., & Kaye, W.S. (in press, 2009). Review of Understanding psychology as a science: An introduction to scientific and statistical inference. British Journal of Mathematical & Statistical Psychology.

Dienes, Z. (2008). Understanding Psychology as a Science: An Introduction to Scientific and Statistical Inference. Basingstoke: Palgrave Macmillan.

Chi-square test of independence and the odds ratio

2009-10-20T10:52:00.020+01:00

Every time I teach an introductory statistics course I'm struck by how difficult it is to run a simple 2 by 2 chi-square test of independence. SPSS is one of the worst culprits (but even in R it almost seems more trouble than it is worth to use a computer). One solution is calculate it by hand (and that's what I do in my introductory classes). This leaves me with a practical problem - how can I quickly and easily calculate solutions to practical exercises, check student work or calculate chi-square statistics for my own work?

I've mostly assumed that my solution is so obvious that it is what most statistics teachers do. I just set up an Excel spreadsheet that calculates the chi-square statistic, degrees of freedom and p value for a 2 by 2 table. Thus all I have to do is type in four numbers.

Over the years I've included a few other features - it displays the intermediate calculations (useful for checking student work), calculates standardized residuals and various common 'effect size' estimates. These are phi, phi-squared and the odds ratio. For instance, I also added the Haldane estimator of the odds ratio (which is the odds ratio estimated after adding 0.5 to each cell). This is a useful estimator when observed counts are zero or very small.

My favourite application of the spreadsheet is when editing or reviewing journal submissions reporting chi-square. Often they will not consider the odds ratio. The spreadsheet means I can check calculations for accuracy (if they look dubious) and I can easily include odds ratios in my review or decision (all in under a minute). The odds ratio is useful because it strips out the base rates when comparing effects from different conditions.

Using the absolute difference in rates for chi-squares with different base rates can be misleading. For instance, the difference in solution rates in an easy problem solving task might be 70% - 50% = 20%. For a very hard problem it might be 20% - 10% = 10%. Comparing the absolute differences in rates (often called the ARR or absolute risk reduction in medical settings) would be misleading. The 10% difference is probably more impressive for the very hard problem (it represents a doubling or halving of solution rates). The odds ratio allows a comparison of the probability of solution relative to the probability of non-solution (i.e., the odds) and thus strips out the base rate impact (by scaling in terms of the probability of non-solution). The odds ratio also has other nice mathematical properties (which I won't go into here).

Not everyone loves odds ratios. Medics often dislike them ... because they strip out base rates! For interpreting and communicating medical risks the base rates are important. It matters whether a disease has a base rate of 1 in 100 or 1 in 10. So stripping out the base rate might be misleading in this context. Even so, medics seem unreasonably biased against odds ratios. If the base rates in your sample are dodgy (i.e., don't reflect the population you are interested in) the odds ratio is probably a safer bet. If you know the odds ratio and the correct base rates you can estimate the true risks by combining the two.

(This is one reason why I think that the quest for the perfect effect size statistic is flawed. Different statistics are required for different jobs. Odds ratios, however, get a (largely undeserved) bad press. Most psychologists will probably be better off using them than other measures for 2 by 2 tables, because we rarely have samples that accurately sample the 'true' base rate. This is because our samples are either unrepresentative or, even more importantly, there may be no 'true' base rate. The problem solving example illustrates this. A problem doesn't really have a 'true' difficulty level that we are trying to generalize to or make decisions about.)

If you want to use the spreadsheet you can download it from here. Alternatively, just set up your own (it is quite a useful exercise for understanding how chi-square, odds ratios and so on work). On the other hand my spreadsheet does a few of the slightly awkward calculations for you (e.g., CIs for the odds ratios). The first sheet has very basic instructions. The second sheet has the 2 by 2 table calculations (and effect sizes). The other sheets provide basic statistics for 2 by 3 and 3 by 3 tables. No effect size metrics are included for the latter cases because I think they are not very meaningful for effects with df > 1 (and I've never needed them for real research).

Note: for best results (to preserve formatting etc.) download the document as an Excel file.

Don't standardize interaction/moderator effects in multiple regression

2009-03-12T19:13:00.005Z

This is another quick blog entry related to a query I had today.

In my moderated multiple regression workshop a while back I wrote (on slide 15) "don't use standardized regression coefficients". In the talk itself I briefly mentioned why and directed people to the relevant section of the (excellent) explanation by Kris Preacher:

A primer on interaction effects in multiple linear regression.

I'd like to clarify why standardization is a particularly bad idea in this case. My dislike of 'standardization' is fairly well known, and it goes without saying (I hope) that one reason not to use standardized regression coefficient relates to this. I hope to write about this in detail soon (but check out my BJP article on effect size for further details).

However, there are additional reasons why standardizing predictors will cause trouble in moderated multiple regression. Standardization involves centering the predictors and scaling them terms of their sample SDs. Centering is very often a useful thing to do in moderated multiple regression. However, statistics packages such as SPSS will standardize all the predictors - including the product terms - in moderated multiple regression. This is because they have no way of knowing that the product term is not a 'regular' predictor (similarly if anyone were foolish enough to do a stepwise regression ... the software would not know to keep in X1 and X2 for each X1.X2 product term). This means that the X1.X2 product term will be standardized along with X1 and X2 rather than being computed (correctly) as the product of the two standardized predictors (i.e., Zx1 and Zx2 multiplied together).

That's clearly a problem (the t test should be OK but the value of the coefficient and simple slopes will be wrong). However, the clincher is that even if the correct standardization is carried out (e.g., computing the standardized predictors yourself and then taking the product of the relevant standardized predictors and entering it into the regression) the standard errors will be incorrect (which is problematic for constructing confidence and prediction intervals).

In summary, you can't rely on the software to get the standardization correct so use the unstandardized regression coefficients and standard errors! (Also, did I mention that standardized coefficients are generally a bad idea anyway?).

R resources for psychologists

2009-01-12T23:07:00.004Z

As a quick follow-up to my previous post, I'll quickly note some R links specifically aimed at psychologists. Jonathan Baron and Yuelin Li have a excellent set of notes that cover ground from R basics to ANOVA (including repeated measures ANOVA) and beyond. Jonathan Baron also maintains a general set of R links. William Revelle's R guide for psychological research is also very good and has more emphasis on psychometrics than Baron and Li.

Pilgrim's blog is an R resource for the exercises in the classic Applied Multiple Regression/Correlation Analysis for the Behavioral Sciences text.

Simulating data for inquiry based learning

2009-01-09T19:32:00.021Z

I've just presented a talk on at the Statistics for Psychology Students workshop for the HEA Psychology Network in York. Richard Rowe (Psychology, Sheffield) gave an interesting talk on teaching statistics via inquiry based learning. Part of the work involved using getting students to generate their own research questions in a tutorial and then analyzing data addressing these questions in a follow-up tutorial. A rather clever idea he reported was to generate suitable data via simulation in STATA. The students were first or second year undergraduates and so the experimental designs they came up with were constrained to simple cases (e.g., t tests, correlations and chi-square for the first years; one-way ANOVA with 3 levels for the second years). This meant that whoever led the tutorial (e.g., a Ph.D. student) could generate tailored data to match the students' research designs.

The novel and clever aspect of this isn't simulating real data sets or getting students to come up with their own research ideas, but combining the two. This means that you can run these sessions without getting students to collect real data. (There is a place for that too, but collecting real data has massive overheads in terms of student time, staff time, ethics approval and so forth). Furthermore, students can legitimately come up with ideas for studies that can't be run by undergraduates for ethical, resource or other reasons - for example using clinical populations). I can also see other uses for this. For example, a supervisor could simulate data for a final year project and a student could use the simulated data as a dry run for the real analysis. (The real data will usually be much messier, but I think that getting familiar with the statistical software and analyses to be used could be useful for some students).

A few members of the audience asked questions about STATA (which I'm not familar with) and I pointed out that you could do the same things in R for free. As I understand it STATA allows you to simulate data with a specified covariance matrix fairly easily. I'm sure this can be done in R too, but I'm still learning how to use R and tracking down the right package and commands would have taken a bit of time. In any case, for this to be useful it needs to be very simple and easy to run by people with relatively basic statistical computing skills (and who have never used R before). So I set myself a challenge of writing code that should run with functions from the R base package, be trivial to edit and generate usable data for two common analyses: the independent t test and a bivariate correlation.

It took me about 2 or 3 minutes to write the independent t test code. (This may sound impressive but a competent R programmer could probably do it in under a minute). It took a little longer to comment it and work out how to write it to a tab delimited text file. It took me about 5 or 10 minutes to do the bivariate correlation and a little longer to fix a rather stupid error I'd made. All the code is rather clumsily written (and in some cases I've deliberately separated out steps to make it easier for someone unfamiliar with R to follow). The correlation solution (in particular) isn't very good and specifying a covariance matrix to constrain a simulation would be much more satisfactory. (I should also note that it uses a trick I picked up some time ago from Paul Barrett's web site in his article on Correlation attenuation due to Likert categorization.) The key point is that it took longer to describe here than it took to write. I'm sure better solutions exist in R, but these ones work and could easily be extended to paired t tests or one-way ANOVA. (I'm not sure about chi-square. I could write something, but it might be a lot easier just to specify a 2x2 or 3x2 contingency table by hand).

Running the R code is also easy. R runs on PC, Linux and Mac OS X, but if you don't fancy installing it (for some bizarre reason) it can also be run from a web server (though it won't be able to write to a file). To run the code you just paste it into the R console and hit return. To tailor the code just read the comments and tweak the parameter values before pasting.

If you want a copy to play with just email me.

For anyone who wants to do some simulation in R there are lots of resources around (try googling the obvious keywords), but Andrew Gelman and Jennifer Hill's book Data Analysis Using Regression and Multilevel/Hierarchical Models (Analytical Methods for Social Research) is an impressive book for a serious take on using simulation (although the stance is Bayesian this doesn't get in the way - at least for the sections I've read; I'm about 40% through the book).

Postscript. All the talks were video recorded and Anne Trapp threatened to put video podcasts - vodcasts? - on the Psychology Network site at some point in the future - including my talk (Effect size: why what we teach psychology students is wrong). These will be worth looking out for just to see Andy Field's talk. (Andy claims he looks particularly ridiculous when video recorded. My advice: just don't ever look at the recording. Ever.) I'll write something about effect size in the near future - though much of the talk was based on my forthcoming BJP paper on effect size.