Psychological Statistics

Calculating and graphing within-subject confidence intervals for ANOVA

2011-10-04T15:43:00.006+01:00

Psychologists are gradually coming round to the view that it is a good idea to present interval estimates alongside point estimates of statistics. The most common statistic reported in psychology research is almost certainly the mean (strictly the arithmetic mean). Presenting an interval estimate for the mean of a single sample is usually quite simple. This is usually done as 95% confidence interval about the mean – and most researchers in psychology are able to calculate this by hand or get their statistical software to calculate and graph it for them.

Extending this to more than one mean introduces an additional layer of complexity. This is because the difference between two means is a different quantity, and its CI (although related to those of the individual means) is different in width from the CIs of the individual means. This creates a problem when plotting the CI because a researcher might be interested in the CI for an individual mean, the CI for their difference (or both).

The complexity increases further if the aim is to plot a set of means (e.g., from an ANOVA design). In this case, plotting all the possible differences (as is commonly done) obscures patterns in the individual means (e.g., linear or quadratic trends). Last, but not least, if the means are not from independent samples, there are further difficulties. This happens in within-subjects or repeated measures designs.

In these designs the variation around each mean is correlated with the variation around the other means. This correlation arises from individual differences. Statistical procedures such as ANOVA can capitalize on these individual differences to produce more sensitive statistical inferences (i.e., to increase statistical power or obtain narrower CIs). This is done by estimating the variation due to individual differences, and removing it from the error variance (the estimate of statistical noise in the data set).

This is a problem for graphical presentation of means because the precision of individual means is influenced by individual differences, whereas the precision of differences between means is not (because the estimate of individual differences is common to repeated samples from the same people and thus can be removed). Further complications arise when the sphericity assumption of repeated measures ANOVA is violated.

Several solutions to these problems have been proposed in the literature. The best known of these in psychology is that of Loftus and Masson (1994). Another well-known solution is that of Goldstein and Healy (1995), extended to correlated samples by Afshartous and Preston (2010).

Despite a large literature on the problems of graphing a set of correlated means, many people avoid the problems altogether by not reporting (or graphing) CIs or report CIs that are misleading in some way. Researchers are often unaware of the problems or find the solutions hard to understand and implement.

I recently reviewed the main approaches in the literature, describe how to obtain suitable intervals for individual means and differences between means and provide R code to calculate and plot the intervals.

The main highlights are that:

i) for inferences about individual means the standard approach works fairly well for between-subject (independent measures) designs, but there is a case to use CIs from a multilevel model for within-subject (repeated measures) designs

ii) an approach proposed by Cousineau (2005) with a correction by Morey (2008) offers advantages over the Loftus and Masson (1994) approach for within-subject ANOVA designs. It simplifies the calculations and does not assume sphericity. The Loftus-Masson approach will however usually be superior when n is small.

iii) if you are interested in differences between means then you should probably plot a version of the Cousineau-Morey (or Loftus-Masson) interval that is adjusted so that overlap of the CIs around two individual means corresponds to overlap of the CI for their difference. This can be done by incorporating a multiplier to the width of the individual CIs. This multiplier is equal to (2^0.5)/2.

iv) if you are interested in both precision of individual means and their differences you can use a two-tiered error bar to display both quantities (Cleveland, 1985).

v) the intervals (and graphical presentation of means) are useful for informal inference about a set of means. For formal inference it is better to set up precise hypotheses and test these via an a priori of contrast. This could be a traditional null hypothesis significance test, but other approaches are available. These include confidence intervals, Bayes factors, likelihood ratios and so forth (Baguley, in press; Dienes, 2008).

The paper is available here, the R code here and the data sets here.

References

Afshartous D., & Preston R. A. (2010). Confidence intervals for dependent data: equating nonoverlap with statistical significance. Computational Statistics and Data

Analysis. 54, 2296-2305.

Baguley, T. (2011, in press). Calculating and graphing within-subject confidence intervals for ANOVA. Behavior Research Methods. DOI: 10.3758/s13428-011-0123-7

Baguley, T. (2012, in press). Serious Stats: A guide to advanced statistics for the behavioral sciences. Basingstoke: Palgrave.

Cleveland. W. S. (1985). Elements Of Graphing Data. New York, NY: Chapman & Hall.

Cousineau, D. (2005). Confidence intervals in within-subject designs: A simpler solution to Loftus and Masson’s method. Tutorials in Quantitative Methods for Psychology, 1, 42-45.

Dienes, Z. (2008). Understanding Psychology as a Science: An Introduction to Scientific and Statistical Inference. Basingstoke: Palgrave Macmillan.

Goldstein, H., & Healy, M. J. R. (1995). The graphical presentation of a collection of means. Journal of the Royal Statistical Society. Series A (Statistics in Society), 158, 175-177.

Loftus, G. R., & Masson, M. E. J. (1994). Using confidence intervals in within-subject designs. Psychonomic Bulletin & Review, 1, 476-490.

Morey, R. D. (2008). Confidence intervals from normalized data: A correction to Cousineau (2005). Tutorials in Quantitative Methods for Psychology, 4, 61-64.

A problem of significance

2011-09-15T23:27:00.005+01:00

Several people have drawn my attention to a recent article on a common error in published statistical analyses in neuroscience. Sander Nieuwenhuis, Birte Forstmann and Eric-Jan Wagenmakers published (in Nature Neuroscience) a critique of statistical analyses in the neuroscience literature. This paper has been written about by Ben Goldacre and Andrew Gelman (who published an article on the general problem some time ago) - so I won't go into too much detail.

The point of interest for me is that the error concerns something that most psychologists should know all about (and hence should be expected not to make the error). It concerns the case of two differences, one statistically significant and one non-significant. For example, group 1 may show a significant difference between experimental condition and placebo (for a drug intervention), while group 2 do not. A naive interpretation is that the drug works for group 1 but not group 2. This is not necessarily true. The proper test of a difference in effects of the drug between groups is an interaction test. Psychologists tend to avoid this error because we have heavily trained in ANOVA as undergraduates (certainly in the UK and probably also in the US and most of Europe). Even if we fail to learn this, reviewers and editors (in psychology) tend to spot the error.*

Are psychologists then entitled to feel a little bit smug? Perhaps, but only a little. First, I think the reason we are relatively good performers on this point is because we tend to view many statistical analyses through an "ANOVA" lens. Factorial ANOVA (in which factors are orthogonal) includes the interaction term by default. The 2 by 2 factorial ANOVA is the workhorse of experimental psychology. Our familiarity with this type of design and analysis makes this easy to spot. Second, our ANOVA lens leads to other errors - notably dichotomizing continuous variables (e.g., via median split) in order to squeeze them into an ANOVA design. This always decreases statistical power, and can - albeit infrequently - produce spuriously significant effects (see MacCallum et al., 2002). These errors are sometimes less serious than the difference of differences/interaction error (but are not harmless).

The real test then, is whether psychologists make the same (conceptual) error in a different context. The obvious context is that of association rather than difference. If males show a significant correlation between testosterone and aggression (e.g., r = .5, N = 25) and females don't (e.g., r = .3, N = 25), the correlation between testosterone and aggression is not significantly bigger for males than females. To confirm this you'd need to construct a test or (better still) confidence interval for the difference in correlations. This is hardly ever done - and, in my experience, psychologists frequently make this kind of claim without backing it up.** Methods for testing differences in correlations are a bit fiddly (e.g., depending on overlap or lack of overlap in the measurements), and rarely taught at undergraduate or even postgraduate level. The methods that are taught are also often a bit dodgy (see Zou, 2007, for some better alternatives).

Also note that (in both cases) the error can work the other way. Two correlations could be non-significantly different from zero but different from each other (e.g., r = .5 and r = -.5 with N = 10).

Postscript

There is, I think, a lesson or two here. A minor lesson is that interactions are bit more complicated than psychologists (particularly those very familiar with ANOVA) often think. I could write more on this (and do a bit in my forthcoming book). A major lesson is that this concept (the difference between significant and non-significant is not necessarily also statistically significant - see Gelman & Stern, 2006) is probably quite tricky. It may be worth exploring why ... I suspect it is because of several factors.

References

Gelman, A., & Stern, H. (2006). The difference between “significant” and “not significant” is not itself statistically significant. American Statistician, 60, 328–331.

MacCallum, R. C., Zhang, S., Preacher, K. J., & Rucker, D. D. (2002). On the practice of dichotomization of quantitative variables. Psychological Methods, 7, 19-40.

Nieuwenhuis, S., Forstmann, B. U., & Wagenmakers, E.-J. (2011). Erroneous analyses of interactions in neuroscience: A problem of significance. Nature Neuroscience, 14, 1105-1107.

Zou, G. Y. (2007). Toward using confidence intervals to compare correlations. Psychological Methods, 12, 399-413.

* Do I have any support for this position? Yes: anecodotal support (e.g., from editing or reviewing many dozens of papers) and some support from Nieuwenhuis et al. They found the error more prevalent in cellular and molecular neuroscience. ANOVA is core training in psychology and widely used in cognitive and behavioural neuroscience - and I'd argue that this reflects the influence of psychologists working in this area and of neuroscientists trained in and using similar methods.

** Do I have any support for this position? A little. It is easy to find basic psychology texts with ANOVA but without tests of differences in correlations being mentioned. It is rare to find tests of CIs of differences in published papers.

R: An introduction for psychologists

2011-01-13T13:50:00.000Z

Here are the slides from the Introduction to R session Danny Kaye and I ran at the BPS Mathematics, Statistics & Computing section CPS Workshop (13 December 2010, Nottingham Trent University).

Introduction to using R in research

2010-05-13T16:21:00.000+01:00

I was recently asked to give a talk to our graduate school annual conference. I offered several titles and the one they picked was Using R in research. I'm not sure if this was a good idea or not. The graduate school covers PhD students across three areas of the university: social sciences (including psychology), law and business. In addition the students tend to specialize in either qualitative or quantitive research methods, so I was talking to an audience who might know nothing about statistics or a great deal (e.g., several students have completed MSc courses in psychological research methods here or elsewhere).

My solution was try and explain the advantages of R relative to alternatives such as SPSS (probably the most common statistic package in the University). I also focussed a lot on graphical methods and simulation. It seemed to go quite well, but I worry that quite a few members of the audience were overwhelmed by large chunks of it.

I promised to put my slides on my blog - though I am not sure how useful they are to anybody who wasn't there. Without my commentary some (possibly most) of the slides won't make much sense. I spent a good deal of the time talking through exploratory plots of one data set (from Hayden, 2005). I use this example a lot in teaching and it involves a bit of class participating (guessing the origin of the data) - so I won't go into to detail here (lest I spoil it for future students), but you can google the original article if you are curious. I also spent some time on how R works (e.g., object types, assignment, basic modeling, plotting functions). My reasoning was that many of the audience have no familiarity with non-GUI interfaces in software and without explaining the basics of the interface they will not have the faintest clue how R works. For those with some familiarity (e.g., SPSS syntax) the examples were selected to show how powerful R can be for things like exploratory graphics.

Several students ask about resources for learning R. I mentioned some in earlier blog posts, but for psychologists Li and Baron's web resources are a good place to start. The other major resource is probably Quick R, but there are hundreds of other places to look online (depending on what stuff you need most).

Example of plotting a serial position curve in R

2010-03-03T20:14:00.018Z

A while ago I wrote a co-wrote chapter for an introductory psychology text book Essential Psychology: A Concise Introduction. This is a book edited and written by members of the department where I work. My contribution was the chapter on human memory (cunningly titled Memory).

I produced several plots for the chapter (some of which got cut due to severe space restrictions). One that stayed in was a serial position curve. For this plot I used data from Postman and Phillips (1965).

I feel particular proud of this plot because I was just beginning to use and learn R at the time (as opposed to dabbling) and because I had had a really hard time getting hold of the data. I first tried google, but had no joy (for some reason I thought someone would have put the raw data online, as it is a classic study - though maybe I just missed it). Then I searched for alternative data sets (as around that period there were quite a few similar studies). I was probably being too picky, but whatever the reason I had no luck.

It would have been trivial to make up fake data, but that didn't feel right. What I eventually did (and wished I'd done straight away) was print out the original figure and measure all the points by hand. I then entered these values into a spreadsheet and tweaked and remeasured until all the summary statistics matched those in the original paper to about one decimal place. This was a lot quicker than I had thought. I cheated slightly because I only needed data from the 20 word conditions (so I could leave out the 10 and 30 word conditions).

(I'm pretty sure I could have used computer software to capture the raw data from an image file, but I'd have had to find the software, learn how to use it and do all the checking anyway. For a single figure I'm reasonably sure measuring by hand would be faster.)

In re-plotting it I noticed a few things that I hadn't paid much attention to before. The main one was the authors report frequency of recalls for 18 participants with 6 lists each. This means all scores are out of 108 and I suspect lots of casual readers would (like me) assume they were percentages. For re-plotting I rescaled the data as percentages.

The plot itself just uses basic R functions. I'm writing about it because: i) I think it is a fairly clear illustration of how basic plot functions in R can produce what I think is a rather nice Figure. (The published version has been edited by the publisher, adding colour and making the style match figures in other chapters), ii) people may find it useful for teaching purposes. So please feel free to use and adapt the R code for non-commercial (e.g., teaching use).

First load the data from this .csv file (you will need to specify the path or change the working directory if the file is saved elsewhere).

pp65 <- read.csv("pp65.csv")

Then paste the following:

plot(pp65$SP, pch=NA, ylim=c(0,80), xlab= "Serial position", ylab= "Mean percentage recall", main = "Postman & Phillips (1965)", sub = '(20 word conditions only)')

points(pp65$C0, pch=19, col='black', cex=.7)

lines(pp65$C0, lty=3)

points(pp65$C15, pch=24, col='black', cex=.7)

lines(pp65$C15, lty=2)

points(pp65$C30, pch=22, col='black', cex=.7)

lines(pp65$C30, lty=5)

legend(3, 80, legend=c("No delay","15 second delay","30 second delay"), lty=c(3,2,5))

If you are new to R you can find out more about these plotting functions by using R help: ?par, ?plot, ?points and so on ...

References

Baguley, T., & Edmonds, A. J. (2010). Memory. In P. Banyard, M. N. O. Davies, C. Norman, & B. Winder (Eds.) Essential Psychology: A Concise Introduction (pp. 65-82). London: Sage.

Postman, L. & Philips, L. W. (1965). Short-term temporal changes in free recall. Quarterly Journal of Experimental Psychology, 17, 132-138.

Interaction plot from cell means

2010-02-25T00:23:00.001Z

I needed to produce a few a interaction plots for my book in R and, while the interaction.plot() function is useful it has a couple of drawbacks. First, the default output isn't very pretty. Second, it works from the raw data, whereas I often need plots from cell means. For teaching purposes it is quite common to produce plots without raw data (for hypothetical data or from published examples).

My first attempts at the plots involved setting them up element by element. Just going over some examples I decided to turn the basic plot (for a 2 x 2 ANOVA) into a simple function. Nothing fancy, just a regular interaction plot in black and white that I think is prettier than the SPSS, Excel or R defaults. At some point I may have a go turning it into a general I x J ANOVA plot (or maybe even add CIs, but I'll probably do that from raw data if I ever get round to it).

plot.2by2 <- function(A1B1,A1B2, A2B1, A2B2, group.names, legend = TRUE, leg.loc=NULL, factor.labels=c('Factor A', 'Factor B'), swap = FALSE, ylab= NULL, main = NULL){
group.means <- c(A1B1, A2B1, A1B2, A2B2)
if(missing(ylab)) ylab <- expression(italic(DV))
if(swap==TRUE) {
group.names <- list(group.names[[2]], group.names[[1]]) ; group.means <- c(A1B1, A1B2, A2B1, A2B2); factor.labels <- c(factor.labels[2], factor.labels[1])
}
plot(group.means, pch=NA, ylim=c(min(group.means)*.95, max(group.means)*1.025), xlim=c(0.8,2.2), ylab=ylab, xaxt='n', xlab=factor.labels[1], main=main)
points(group.means[1:2], pch = 21)
points(group.means[3:4], pch = 19)
axis(side = 1, at = c(1:2), labels = group.names[[1]])
lines(group.means[1:2], lwd = .6, lty = 2)
lines(group.means[3:4], lwd = .6)
if(missing(leg.loc)) leg.loc <- c(1,max(group.means))
if(legend ==TRUE) legend(leg.loc[1], leg.loc[2],legend = group.names[[2]], title = factor.labels[2], lty = c(3,1))
}

Call the function by entering the four cell means in conventional order: A1B1, A1B2 and so on where A1B1 is the mean of level 1 of factor A at level 1 of factor B. You also need a two item list containing text strings of the two level names of each factor. For instance:

lev.names <- list(c('A1', 'A2'), c('B1', 'B2'))
plot.2by2(5,15,10,20, lev.names)

You can swap the axes by adding the argument swap = TRUE:

plot.2by2(5,15,10,20, lev.names, swap = TRUE)

The default factor names are 'Factor A' and 'Factor B', but these are over-ridden in the call:

plot.2by2(5,15,10,20,lev.names, swap = TRUE, factor.labels= c('Factor 1','Factor 2'))

You can also change the y-axis label with ylab or add a main title with main. The legend can be dropped (legend = FALSE) if you don't want one or need it to be located outside the plot. To move the legend just specify coordinates with an argument such as leg.loc = c(1,10). You can also edit the source code directly. Here is an example with title and meaningful labels:

group.names <- list(c('placebo','drug'), c('male', 'female'))

plot.2by2(10,10,15,20, group.names, factor.labels=c('Drug', 'Sex'), swap = FALSE)

As this just uses basic plotting functions in R you can also manipulate the plot in other ways: adding lines with segments(), adding text with text() changing graphical parameters with par() and so on. Depending on your platform it is also easy to extract the plot as a .pdf or .jpg file. On a mac I save it as a .pdf file and open it in preview which allows me to save it as .png, .gif or whatever I need.

When correlations go bad ... (Or, I always wanted to write about the Society for the Suppression of the Correlation Coefficient)

2010-01-29T16:48:00.035Z

A while ago I promised a longer post on standardized effect size. This isn't it. Instead it is a link to my piece in the February 2010 issue of The Psychologist.

I had intended to write a short summary of my 2009 BJP paper, but that didn't really get off the ground. However, I had for a while wanted to write about the Society for the Suppression of the Correlation Coefficient. I had first read about this in Tukey's (1954) chapter on regression and path analysis. This is one of the earlier papers in the literature criticizing the preference for (standardized) correlation coefficients over (simple, unstandardized) regression coefficients. Read the article for an earlier example! I was reminded of its existence by Brillinger's (2001) paper.

As soon as I tried writing about Society for the Suppression of the Correlation Coefficient things (I think) fell into place. The result is a piece that is one part history of statistics trivia, one part mini-tutorial and one part a summary of my 2009 paper.

For non-members of the BPS the link below contains a pre-publication version of 'When correlations go bad ...'.

I may also get around to the web site one day.

P.S. I adapted data and R code from Gelman and Hill (2007) for the example. I chose it because it a nice simple example of regression and because it is also a pointer to someone who argues in favour of standardization )at least in some situations). Both my 2009 paper and the 'When correlations go bad ...' are my attempts at getting people to rethink the use of correlation and standardization. The tone is deliberately (slightly?) polemic. There are other views so don't just take my word for it ... think about what you are trying to do and decide whether a correlation coefficient (or a standardized mean difference) is the right option.

References

Baguley, T. (2009). Standardized or simple effect size: What should be reported? British Journal of Psychology. 100, 603–617.

Baguley, T. (2010). When correlations go bad … The Psychologist, 23, 122-3.

Brillinger, D.R. (2001). John Tukey and the correlation coefficient, Computing Science and Statistics, 33, 204–218.

Gelman, A. & Hill, J. (2007). Data Analysis Using Regression and Multilevel/Hierarchical Models (Analytical Methods for Social Research). Cambridge: Cambridge University Press.

Tukey, J.W. (1954). Causation, regression and path analysis. In O. Kempthorne, T.A. Bancroft, J.W. Gowen & J.L. Lush (Eds.) Statistics and mathematics in biology (pp.35–66). Ames, IA: Iowa State College Press.

Strathclyde multilevel modeling talk

2010-01-25T09:40:00.003Z

I recently gave an introduction to multilevel modeling talk at Strathclyde University for PsyPAG. I promised to make my slides available. I'm still using powerpoint (and regretting it) so some of the symbols may be garbled. (I noticed that powerpoint turned all my tilda symbols to colons during the talk). So I hope it is readable:

Here is the link:

A statistical puzzle about averages II

2009-12-13T10:26:00.119Z

1. Who is correct? Professor B is correct. If the average family has 1.8 children then the average child would be expected to have more than 0.8 siblings.

2. Why? The average child is not from the average family. The concepts of average child and average family are different. For this reason there should no expectation that the average child should be from a family with an average number of children. Although there are restricted circumstances under which this can happen, they are sufficiently implausible to be discounted in any real world application (e.g., if all families have exactly the same number of children).

In one case the unit of analysis of the family and in the other it is the child. A concrete example may help. (I'll stick to defining average as arithmetic mean throughout, but the same logic extends to other averages such as the median - see the quotation from Kish below).

Assuming that the number of children varies between families (which it must do if the mean number of children per family is 1.8) then the average child will be from a family with a larger number of children than average. For example, imagine there are only four families:

		Number of siblings per child
Family	Number of Children	1^st child	2^nd child	3^rd child	4^th child
A	1	0	-	-	-
B	1	0	-	-	-
C	2	1	1	-	-
D	4	3	3	3	3

There are (1+1+2+4) = 8 children in the four families, thus the mean number of children per family is: 8/4 = 2.

It follows that the mean number of siblings per child is therefore:

(0+0+1+1+3+3+3+3)/8 = 14/8 = 1.75

So although each family has only two children (on average) each child has 1.75 siblings (not 1 sibling).

Note is that there are N = 4 families and n = 8 children. So switching the unit of analysis changes the denominator. Also note that while the families can plausibly be considered independent of each other the children can't (all children in the same family have the same number of siblings in this example, and more generally the number of siblings will be correlated).

What about zeroes? In the calculations above I excluded childless households as families. If you include only households without children as families the discrepancy would be larger.

Does it matter? Yes. Much real world data is clustered in this way. It is important to realize that the average is not going to be from the average (e.g., the average worker isn't from the average firm).

In practical terms this means that careful attention needs to be paid to sampling from families, workplaces, schools and so forth. A random sample of children will disproportionately sample children from large families. There are also social policy implications (e.g., if you are interested in reducing child poverty). Another example is that a random sample of schools will disproportionately sample small schools.

There are also other problems with analysis of clustered data. For this reason anyone working with clustered data needs to seriously consider using multilevel modeling or other methods that i) take into the clustering and ii) allow hypotheses about different levels of the model (e.g., children and families) to be explored.

Postscript

This is quite an old puzzle. I first came across this puzzle in the article by Jenkins and Tuten (1992). They include formulae for deriving one average from the other and cite Huntington (1924) and other later observations of the same phenomenon. I made the connection to multilevel models a little later. For a good (if slightly out of date) introduction to multilevel models see Snijders and Bosker (1999). Recently I noticed that Kish (1965) discusses the same phenomenon in passing.

This quote from Kish sets out the problem quite clearly:

Although the mean number of adults per household is only 2.02, the mean number of household members is 2.24 for the average adult. The greater size ranges of large organizations produce more striking effects. In 1960, 50 million people lived in 130 U. S. cities that had 100,000 or more population; in this population, the average city size was 0.39 million, but the size of the city in which the average person lived was 2.0 millions. Using medians does not help: the median city size was 0.19 million, but the median person lived in a city of 0.62 million.

Kish (1965, p. 571).

References

Jenkins, J. J., & Tuten, J. T. (1992). Why isn’t the average child from the average family? – and similar puzzles. American Journal of Psychology, 105, 517-526.

Kish, L. (1965). Sampling Organizations and Groups of Unequal Sizes, American Sociological Review, 20, 564-72.

Sniders, T. & Boskers, R. (1999). Multilevel Analysis: An Introduction to Basic and Advanced Multilevel Modeling. London: Sage.

A statistical puzzle about averages I

2009-12-09T22:46:00.002Z

I wrote this a few years ago for a departmental newsletter. For some reason the second part (with the answer) never got published. I stumbled across it almost by accident the other day and thought I'd share it. I'll publish the canonical answer in due course.

Professor Quack knows (from the UK census) that the average family has an arithmetic mean of 1.8 children. He also knows that (due to a bizarre mix-up in enrolment) that his Psychology For Everyone class is attended by a random sample of 50 people from the UK population. As part of an in-class demonstration of sampling theory he records the number of siblings of each student and calculates the average (using the standard formula for the arithmetic mean).

To his dismay he discovers that the students in his class have an arithmetic mean of 1.2 siblings. He later repeats the demonstration with two more random samples of the UK population (again with n = 50) and obtains values of 1.1 and 1.3 siblings.

Profesor Quack consults two of his colleagues: Professor A and Professor B.

Professor A replies thus:

“You are right to be dismayed. Bias has somehow entered either your sampling procedure or your calculation of the mean. The true mean number of siblings should be 1.8 - 1 = 0.8.”

Professor B interjects thus:

“Nonsense! All is as it should be. The expected number of siblings in a random sample is most certainly not 0.8. Rather, one would expect the average student to have more than 0.8 siblings, just as you have observed.”

1) Who is correct? Professor A or Professor B?

2) Why?

R functions for Dienes (2008) Understanding Psychology as a Science

2009-11-17T09:43:00.016Z

I recently wrote a review of Understanding psychology as a science: an introduction to scientific and statistical inference by Zoltan Dienes (2008). Dienes' book covers Neyman-Pearson null hypothesis significance testing, Bayesian inference and the likelihood method of inference (inspired by Fisher and associated with A. W. F. Edwards and more recently R. Royall).

One of the most useful features of the book is that Dienes provides Matlab code for examples of calculations in the book (e.g., for Bayes factors, likelihood intervals and so forth). This is not so useful for me because I don't use Matlab. Matlab licenses are also quite expensive and may not be possible for students to access it in many Psychology departments. For those without access to Matlab, Dienes also provides calculators for a number of functions on his own web page for the book. (The calculators are found by following the links to the appropriate chapter, so the Bayes factor calculator is found by following the Chapter 4 link).

Danny Kaye and I thought it would be useful to write R code to compliment the Matlab code for Dienes' functions as a 'bonus feature' for the review. As these functions and the notes for them take up quite a bit of space we decided to include only one, for a Bayes factor, in the review itself (with some notes on how to use it). Danny did most of the work writing functions, which are more-or-less direct translations of the original Matlab code (and have been checked against the web versions). The full set of functions is hosted on his web site along with the notes on how to use them. Also included are page references for the examples in the book.

Why did we write the R functions? First, they offer convenient access to the functions for teachers and students (because R is free and runs on Windows, Mac OS or Linux operating systems). Second, it reduces the burden on Dienes' web calculator (at a marginal decrease in ease of use). Third, R is open source so it is simple to see how the code works and to edit, extend or adapt it (though it is polite to acknowledge the authors of the original code). Fourth, we want to encourage more people to start using R!

As an example, I've already written some alternative functions for likelihood intervals (though as I happened I re-wrote these almost from scratch to get them to plot the likelihood function and interval and to take advantage of some built-in R functions). Those functions are intended for a the book I'm working on and so should appear in due course.

For those who are interested Danny and I are presently working on implementing Bayesian t tests in R (Bayes factors with objective priors) in a user-friendly way for researchers.

References:

Baguley, T., & Kaye, W.S. (in press, 2009). Review of Understanding psychology as a science: An introduction to scientific and statistical inference. British Journal of Mathematical & Statistical Psychology.

Dienes, Z. (2008). Understanding Psychology as a Science: An Introduction to Scientific and Statistical Inference. Basingstoke: Palgrave Macmillan.

Chi-square test of independence and the odds ratio

2009-10-20T10:52:00.020+01:00

Every time I teach an introductory statistics course I'm struck by how difficult it is to run a simple 2 by 2 chi-square test of independence. SPSS is one of the worst culprits (but even in R it almost seems more trouble than it is worth to use a computer). One solution is calculate it by hand (and that's what I do in my introductory classes). This leaves me with a practical problem - how can I quickly and easily calculate solutions to practical exercises, check student work or calculate chi-square statistics for my own work?

I've mostly assumed that my solution is so obvious that it is what most statistics teachers do. I just set up an Excel spreadsheet that calculates the chi-square statistic, degrees of freedom and p value for a 2 by 2 table. Thus all I have to do is type in four numbers.

Over the years I've included a few other features - it displays the intermediate calculations (useful for checking student work), calculates standardized residuals and various common 'effect size' estimates. These are phi, phi-squared and the odds ratio. For instance, I also added the Haldane estimator of the odds ratio (which is the odds ratio estimated after adding 0.5 to each cell). This is a useful estimator when observed counts are zero or very small.

My favourite application of the spreadsheet is when editing or reviewing journal submissions reporting chi-square. Often they will not consider the odds ratio. The spreadsheet means I can check calculations for accuracy (if they look dubious) and I can easily include odds ratios in my review or decision (all in under a minute). The odds ratio is useful because it strips out the base rates when comparing effects from different conditions.

Using the absolute difference in rates for chi-squares with different base rates can be misleading. For instance, the difference in solution rates in an easy problem solving task might be 70% - 50% = 20%. For a very hard problem it might be 20% - 10% = 10%. Comparing the absolute differences in rates (often called the ARR or absolute risk reduction in medical settings) would be misleading. The 10% difference is probably more impressive for the very hard problem (it represents a doubling or halving of solution rates). The odds ratio allows a comparison of the probability of solution relative to the probability of non-solution (i.e., the odds) and thus strips out the base rate impact (by scaling in terms of the probability of non-solution). The odds ratio also has other nice mathematical properties (which I won't go into here).

Not everyone loves odds ratios. Medics often dislike them ... because they strip out base rates! For interpreting and communicating medical risks the base rates are important. It matters whether a disease has a base rate of 1 in 100 or 1 in 10. So stripping out the base rate might be misleading in this context. Even so, medics seem unreasonably biased against odds ratios. If the base rates in your sample are dodgy (i.e., don't reflect the population you are interested in) the odds ratio is probably a safer bet. If you know the odds ratio and the correct base rates you can estimate the true risks by combining the two.

(This is one reason why I think that the quest for the perfect effect size statistic is flawed. Different statistics are required for different jobs. Odds ratios, however, get a (largely undeserved) bad press. Most psychologists will probably be better off using them than other measures for 2 by 2 tables, because we rarely have samples that accurately sample the 'true' base rate. This is because our samples are either unrepresentative or, even more importantly, there may be no 'true' base rate. The problem solving example illustrates this. A problem doesn't really have a 'true' difficulty level that we are trying to generalize to or make decisions about.)

If you want to use the spreadsheet you can download it from here. Alternatively, just set up your own (it is quite a useful exercise for understanding how chi-square, odds ratios and so on work). On the other hand my spreadsheet does a few of the slightly awkward calculations for you (e.g., CIs for the odds ratios). The first sheet has very basic instructions. The second sheet has the 2 by 2 table calculations (and effect sizes). The other sheets provide basic statistics for 2 by 3 and 3 by 3 tables. No effect size metrics are included for the latter cases because I think they are not very meaningful for effects with df > 1 (and I've never needed them for real research).

Note: for best results (to preserve formatting etc.) download the document as an Excel file.

Don't standardize interaction/moderator effects in multiple regression

2009-03-12T19:13:00.005Z

This is another quick blog entry related to a query I had today.

In my moderated multiple regression workshop a while back I wrote (on slide 15) "don't use standardized regression coefficients". In the talk itself I briefly mentioned why and directed people to the relevant section of the (excellent) explanation by Kris Preacher:

A primer on interaction effects in multiple linear regression.

I'd like to clarify why standardization is a particularly bad idea in this case. My dislike of 'standardization' is fairly well known, and it goes without saying (I hope) that one reason not to use standardized regression coefficient relates to this. I hope to write about this in detail soon (but check out my BJP article on effect size for further details).

However, there are additional reasons why standardizing predictors will cause trouble in moderated multiple regression. Standardization involves centering the predictors and scaling them terms of their sample SDs. Centering is very often a useful thing to do in moderated multiple regression. However, statistics packages such as SPSS will standardize all the predictors - including the product terms - in moderated multiple regression. This is because they have no way of knowing that the product term is not a 'regular' predictor (similarly if anyone were foolish enough to do a stepwise regression ... the software would not know to keep in X1 and X2 for each X1.X2 product term). This means that the X1.X2 product term will be standardized along with X1 and X2 rather than being computed (correctly) as the product of the two standardized predictors (i.e., Zx1 and Zx2 multiplied together).

That's clearly a problem (the t test should be OK but the value of the coefficient and simple slopes will be wrong). However, the clincher is that even if the correct standardization is carried out (e.g., computing the standardized predictors yourself and then taking the product of the relevant standardized predictors and entering it into the regression) the standard errors will be incorrect (which is problematic for constructing confidence and prediction intervals).

In summary, you can't rely on the software to get the standardization correct so use the unstandardized regression coefficients and standard errors! (Also, did I mention that standardized coefficients are generally a bad idea anyway?).

R resources for psychologists

2009-01-12T23:07:00.004Z

As a quick follow-up to my previous post, I'll quickly note some R links specifically aimed at psychologists. Jonathan Baron and Yuelin Li have a excellent set of notes that cover ground from R basics to ANOVA (including repeated measures ANOVA) and beyond. Jonathan Baron also maintains a general set of R links. William Revelle's R guide for psychological research is also very good and has more emphasis on psychometrics than Baron and Li.

Pilgrim's blog is an R resource for the exercises in the classic Applied Multiple Regression/Correlation Analysis for the Behavioral Sciences text.

Simulating data for inquiry based learning

2009-01-09T19:32:00.021Z

I've just presented a talk on at the Statistics for Psychology Students workshop for the HEA Psychology Network in York. Richard Rowe (Psychology, Sheffield) gave an interesting talk on teaching statistics via inquiry based learning. Part of the work involved using getting students to generate their own research questions in a tutorial and then analyzing data addressing these questions in a follow-up tutorial. A rather clever idea he reported was to generate suitable data via simulation in STATA. The students were first or second year undergraduates and so the experimental designs they came up with were constrained to simple cases (e.g., t tests, correlations and chi-square for the first years; one-way ANOVA with 3 levels for the second years). This meant that whoever led the tutorial (e.g., a Ph.D. student) could generate tailored data to match the students' research designs.

The novel and clever aspect of this isn't simulating real data sets or getting students to come up with their own research ideas, but combining the two. This means that you can run these sessions without getting students to collect real data. (There is a place for that too, but collecting real data has massive overheads in terms of student time, staff time, ethics approval and so forth). Furthermore, students can legitimately come up with ideas for studies that can't be run by undergraduates for ethical, resource or other reasons - for example using clinical populations). I can also see other uses for this. For example, a supervisor could simulate data for a final year project and a student could use the simulated data as a dry run for the real analysis. (The real data will usually be much messier, but I think that getting familiar with the statistical software and analyses to be used could be useful for some students).

A few members of the audience asked questions about STATA (which I'm not familar with) and I pointed out that you could do the same things in R for free. As I understand it STATA allows you to simulate data with a specified covariance matrix fairly easily. I'm sure this can be done in R too, but I'm still learning how to use R and tracking down the right package and commands would have taken a bit of time. In any case, for this to be useful it needs to be very simple and easy to run by people with relatively basic statistical computing skills (and who have never used R before). So I set myself a challenge of writing code that should run with functions from the R base package, be trivial to edit and generate usable data for two common analyses: the independent t test and a bivariate correlation.

It took me about 2 or 3 minutes to write the independent t test code. (This may sound impressive but a competent R programmer could probably do it in under a minute). It took a little longer to comment it and work out how to write it to a tab delimited text file. It took me about 5 or 10 minutes to do the bivariate correlation and a little longer to fix a rather stupid error I'd made. All the code is rather clumsily written (and in some cases I've deliberately separated out steps to make it easier for someone unfamiliar with R to follow). The correlation solution (in particular) isn't very good and specifying a covariance matrix to constrain a simulation would be much more satisfactory. (I should also note that it uses a trick I picked up some time ago from Paul Barrett's web site in his article on Correlation attenuation due to Likert categorization.) The key point is that it took longer to describe here than it took to write. I'm sure better solutions exist in R, but these ones work and could easily be extended to paired t tests or one-way ANOVA. (I'm not sure about chi-square. I could write something, but it might be a lot easier just to specify a 2x2 or 3x2 contingency table by hand).

Running the R code is also easy. R runs on PC, Linux and Mac OS X, but if you don't fancy installing it (for some bizarre reason) it can also be run from a web server (though it won't be able to write to a file). To run the code you just paste it into the R console and hit return. To tailor the code just read the comments and tweak the parameter values before pasting.

If you want a copy to play with just email me.

For anyone who wants to do some simulation in R there are lots of resources around (try googling the obvious keywords), but Andrew Gelman and Jennifer Hill's book Data Analysis Using Regression and Multilevel/Hierarchical Models (Analytical Methods for Social Research) is an impressive book for a serious take on using simulation (although the stance is Bayesian this doesn't get in the way - at least for the sections I've read; I'm about 40% through the book).

Postscript. All the talks were video recorded and Anne Trapp threatened to put video podcasts - vodcasts? - on the Psychology Network site at some point in the future - including my talk (Effect size: why what we teach psychology students is wrong). These will be worth looking out for just to see Andy Field's talk. (Andy claims he looks particularly ridiculous when video recorded. My advice: just don't ever look at the recording. Ever.) I'll write something about effect size in the near future - though much of the talk was based on my forthcoming BJP paper on effect size.

Beyond ANOVA: from repeated measures to multilevel models

2008-04-09T20:09:00.003+01:00

I have just finished giving a talk on repeated measures ANOVA and multilevel models at the PsyPAG Postgraduate Mathematics, Statistics and Computing workshop. I promised I would upload a link to my presentation. Here it is:

I managed to correct a couple of minor errors - though the transfer online seems to have added several more (though these disappear - or rather the errors mutate - if I swap browsers). I'll add further links and corrections if I get time.

This is run every two years or so in conjunction with the Mathematics, Statistics and Computing section of the British Psychological Society (BPS). The 2008 workshop was at Nottingham Trent University. The full schedule is here:

http://www.psypag.co.uk/msprogramme.html

The statistics clinic was fun - as was my confrontation with Mark Shevlin (of which more later) - but I missed the signed statistics text book raffle (proceeds to charity). I was very interested in John Reidy's talk on moderated multiple regression. Unlike my talk he covered mediation as well, but possibly covered moderation in less detail as a consequence. I think anyone who found John's talk useful will probably want to look at my moderated multiple regression blog entry if only for the link to Kris Preacher's web resources.

Mark Shevlin and I had a (friendly) dispute over the relative merits of SEM and multilevel models and MPLUS versus MLwiN for the latter. For experimental psychologists I think multilevel models are definitely the way to go - it being easy to extend classical experimental designs to multilevel models. For non-experimentalists who will have messier studies to analyze SEM is probably more useful. Although all multilevel models are probably just SEM in disguise I think SEM is harder to pick up and (at least until recently) was not great for repeated measures and clustered data. As for software ... because MLwiN is free for UK academic users I think it would be my top tip for anyone in the UK learning multilevel modeling (with R a very close runner up).

Postscript. The Centre for Multilevel Modeling have just launched their online multilevel modeling course. At first glance it looks very useful, as it starts with simpler statistical techniques (e.g., multiple regression) and builds up to multilevel models. I'm hoping to audit all the modules at some point ... They are released under a creative commons license so it should be possible to use them in Education.

An introduction to moderated multiple regression

2007-10-01T11:24:00.001+01:00

I've just given a workshop on moderated multiple regression as part of a new Psychology Research Methods series at work. Here is the abstract:

Most psychologists are highly familiar with the analysis and interpretation of interaction effects in factorial analysis of variance (ANOVA). In ANOVA these interactions are between categorical independent variables (e.g., experimental conditions). What many researchers do not realize is that interaction effects in ANOVA are a special case of interaction effects in multiple regression which can be computed in much the same way. A common, but unnecessary and inappropriate, strategy for dealing with interaction effects between continuous independent variables is to turn them into categorical ones (e.g., by a median split). A better approach - one with greater statistical power and which is more informative - is to add interaction effects to a multiple regression model. This approach is often known as moderated multiple regression. This workshop will: introduce moderated multiple regression, explain how to compute an interaction term for moderated multiple regression, discuss the role of centring in making the results easier to interpret, and demonstrate how to run and interpret such a moderator analysis in SPSS.

You can view the presentation on google docs:

http://docs.google.com/TeamPresent?fs=true&docid=dp86h6t_3wmr5ft

The workshop is supposed to be a basic introduction. For a more advanced introduction I'd recommend Kris Preacher's web page:

A primer on interaction effects in moderated multiple regression.

Kris Preacher also has links to excellent tools for investigating and plotting moderator effects. (These generate R code that you can plot online with one mouse click or paste into your R workspace to edit for publication quality graphics). Although, for initial exploration it may be quicker to use the regression equation to plug in different X values and see how the expected Y value changes. I'd probably do this by hand (or maybe Excel) and sketch out basic plots by hand.

How to calculate simple main effects using generic ANOVA software

2006-07-10T12:42:00.008+01:00

Some time ago a PhD student of mine needed to conduct simple main effects for several different ANOVA designs. As the explanation of how to do this is a bit fiddly (and it is the kind of thing I get asked about quite a lot) I decided to write the explanation down. This later turned into a web page and now I'm updating it for this blog. The main focus is on explaining how to do the calculations based on output from generic ANOVA software. The calculation methods I describe seem obvious to me, but intended to be the best or most efficient method. Rather, their aim is to make clear what the calculation is doing and to be sufficiently robust that one can adapt them to most (if not all) standard ANOVA software. For any given software there will probably be more efficient solutions (e.g., via syntax in SPSS).

This post aims to set out how to calculate simple main effects for common ANOVA designs. Some statistics packages will calculate simple main effects for you, but many will not do so or will only do so for certain designs. There is also some confusion as to how to calculate simple main effects for some designs. For this reason I aim to set out generic methods to calculate simple main effects that should work for any software package that can cope with common ANOVA designs (even if they don't support direct calculation of simple main effects).

What is a simple main effect?

In analysis of variance (ANOVA) we are often interested only in the effects of a single factor. These effects are termed main effects. A one-way (one factor) ANOVA therefore has only a single main effect. The F test for a main effect tests the hypothesis that the means differ. ¹ In a design with more than one factor (e.g., a two-way design with two factors) we can also look at interaction effects. An interaction between two factors (a two-way interaction) is present if the effects of the factors are not independent. In general researchers will ignore a non-significant
two-way interaction and interpret the two main effects (which are considered to be independent of each other).²

If a two-way interaction is significant it may not make sense to interpret the main effects on their own (because they represent “average” effects of a factor which are known to vary between levels of the other factor).³ Instead, it sometimes a good idea to look at the effect of one factor separately for each level of the other factor. These separate analyses are called simple main effects.

For example, consider a 2 x 2 ANOVA design with gender and age as the factors and anxiety prior to a maths test as the dependent variable. Imagine the means look like this:

	Young	Old
Male	2.2	2.1
Female	3.3	4.5

This pattern shows a fairly clear interaction and also suggests there might be main effects of both age and gender (assuming equal cell sizes).

Let us assume that all three effects are significant. In this example it makes little sense to pay much attention to the main effect of age (because while age does increase anxiety on average, it would be slightly misleading to interpret this as a general effect). On the other hand it doesn't seem unreasonable to interpret the main effect of gender as a general effect (as anxiety seems quite a bit higher for females regardless of age). Nevertheless, to be sure, it seems sensible to look at the effects of age separately for males and females (and possibly also the the effects of gender for young and old participants).

The test of the effect of age on anxiety for male participants is sometimes called the simple main effect of age at "male". The test of the effect of age on anxiety for female participants is called the simple main effect of age at "female".⁴

Note that there are are 4 simple main effects for a 2x2 design, 5 for a 3x2 design and so forth.⁵

The F ratio

As with any ANOVA test we'll want to come up with an F ratio to test the effect.⁶ The F ratio is the ratio of the variance estimate for the treatment effect to the variance estimate of the error. In ANOVA these variance estimates are called mean squares and are calculated by dividing the sums of squares for a source of variation by its degrees of freedom.

The mean square error will depend on whether the factor which is being looked at is independent measures (between subjects) or repeated measures (within subjects). For a mixed design the mean square error depends on which factor is being looked at. For an independent measures factor a pooled error term is used, while for a repeated measures design a non-pooled error term is used. It turns out that obtaining a pooled error term from a mixed ANOVA printout is not trivial so I will return to this later on (see the discussion of pooled errors for mixed ANOVA designs below).

Obtaining the treatment mean square:

One easy way to calculate the treatment mean square is to run separate one-way ANOVAs for each level of the second factor. We can simply read off the mean square value from the print out.

For example, for the simple main effect of gender at young we would merely run a one-way ANOVA with gender as the factor (taking care to exclude all the old participants!).

Obtaining the error mean square (independent measures):

I think the easiest way to obtain the pooled error term used for a pure independent measures design is to read it off the output from the original ANOVA (i.e., the ANOVA with all factors in the analysis). In an independent measures design the pooled error term averages over all the participants and therefore minimizes the effect of individual differences and thus produces the most accurate test.

The appropriate F ratio can be calculated by dividing the treatment mean square by the error mean square in the normal way. This can be checked for significance using tables or by exact methods using software such as Excel or SPSS (both of which have look-up functions for the significance of the central F distribution).

Obtaining the error mean square (repeated measures):

The easiest way to obtain the non-pooled error term required for repeated measures factors is to run separate one-way ANOVAs for each level of the second factor. . A big advantage of this method is that that as the mean square treatment is also obtained this way, the output will give you the correct F ratio and observed significance (p value).

(In some packages, notably SPSS, the simplest ways to do this is are i) copy and paste these data into new columns, or ii) to save the data file under k different names - where k is the number of levels, and delete unwanted levels before running the one-way repeated measures ANOVA. The latter method avoids you having to re-type variable names and other information).

The choice of the non-pooled error term in within-subjects simple effects mirrors the choice in standard repeated measures designs and is now generally accepted as the appropriate procedure.⁷

Obtaining a pooled error term for an independent measures factor from mixed measures ANOVA output:

In a mixed ANOVA design there is at least one repeated measures factor and at least one independent measures factor. Simple main effects for the repeated measures (within subjects)factors should use a non-pooled error term (just as with standard repeated measures design outlined above).

For the independent measures factors we want to use the pooled error estimate from our main analysis. This is problematic because there isn't a single pooled error term in the mixed ANOVA output. Instead, we must take the error terms from the mixed ANOVA output and calculate the pooled error term 'by hand'. Howell (2002, pp. 490-493) describes a (relatively) simple procedure for this.

Calculating the pooled (within cells) SS from the mixed ANOVA output is fairly easy. The first step is to identify the two error terms from the original mixed ANOVA output. These will usually be clearly labelled as error terms.⁸

Designate one as error v and one as error u and combine the sums of squares and mean square as follows:

Pooled SS = SS_u + SS_v

Pooled MS = Pooled SS / (df_u + df_v)

(Where SS stands for sums of squares, MS for mean square and df for degrees of freedom for u or v as appropriate).

So far so good ...

Pooling the sums of squares in this way is fine except that the pooled Mean Square is derived from two different variance estimates. This makes it tricky to work out the correct F distribution for the pooled mean square error. This in turn means that calculating a significance (p) value is tricky. It turns out that we can use the following formula to derive an error d.f. (called f') that gives us the correct p value:

f' = (u + v)² / (u²/df_u + v²/df_v)

[Note: u and v are the SS for the two error terms]

Once you have obtained the pooled MS error simply calculate your simple main effect ratio as normal (F = MS_treatment/MS_error) and evaluate against the usual treatment d.f. and using f' as the error d.f.

While this isn't too difficult it can be a bit fiddly. My preferred solution is to set up a spreadsheet in Excel or a similar program to calculate all these for you (see Resources below).

Looking up significance of F:

Provided you know the F ratio, the treatment df and error df looking up significance is easy. The traditional option is to use tables of significance. These allow you to look up the critical value of F or F_crit that is required to reach significance for a given alpha level. If the observed level of F equals or exceeds F_crit then the test is signifiant at that alpha level. The main drawback of this procedure is that exact significance (p values) can not be obtained (though they can be estimated in some cases).

An alternative strategy is to use a computer program to look up the exact significance (p) value. For example, in Excel one can use the FDIST function.

e.g., FDIST(3.29,1,42) returns the value 0.076851736

Here 3.29 is the observed F ratio, 1 is the treatment d.f. and 42 the error d.f. The p value for this test is therefore marginally non-significant. We could thus report the test as F_1,42 = 3.29, p = .0769.

Resources:

Here is a very basic (and certainly not very pretty) Excel spreadsheet that calculates the pooled MS error used for independent factors in a mixed ANOVA design. The spreadsheet takes as input six numbers (labelled in bold in the spreadsheet). These are SS_u, SS_v, df_u, df_v (the SS and d.f. for the two error terms in the mixed design), the SS_treatment and the df_treatment. The latter two are obtained as normal for simple main effects (e.g., they can be readily taken from the output of separate one-way ANOVAs for each level of a second factor.

Mixed_Pooled_Error.xls

For convenience the spreadsheet also uses the FDIST function to calculate the p value for the simple main effect (using f' as the error d.f. as described above). This function doesn't work in Google Docs but just work just fine in Excel (just select Export from the File menu and save it as an .xls file).

The default values in the spreadsheet are taken from Howell (2002, pp. 492-3).

Bibliography

Howell, D. C. (2002). Statistical Methods for Psychology
(5th. ed.). Pacific Grove, CA: Duxberry.

[David Howell has just released a seventh edition which looks excellent. At first glance the main changes are in layout and use of software examples.]

Footnotes:

1 Strictly, the usual interpretation is that if the F ratio is significant then there is evidence against the null hypothesis that the means are all equal.

2 I have glossed over the finer points of interaction analysis for these pages. There are occasions when non-significant interactions should probably be looked at more closely and occasions when significant interactions should probably be ignored. This relates to the power of the study and the size of the interaction effect. A powerful study may find detect interactions that are negligible in size and a study that lacks power may fail to show significance for an important or large interaction effect.

3 Interpreting main effects in the light of an interaction effect is more of an art than a science. In a mathematical sense interaction effects are independent of main effects and can be interpreted separately. However, verbal interpretations of analyses aren't always sensitive to the mathematics of the situation. The context of the study is also important in the interpretation of the results. If in doubt, always interpret the interaction effect first. Also, if the simple main effects show a consistent pattern (e.g., significant differences or close to significant at all levels of a factor) then the main effects are probably straight-forward to report. In the example given here it probably makes sense to report and interpret the main effect of gender (females are generally more anxious than males in this context), but not age (as it seems misleading to interpret a significant main effect of gender as indicating that older people are generally more anxious in this context).

This decision might differ in another situation (with identical numbers). For example, if the study looked at reading preferences and the factors were gender and font type (serif or sans serif) we might conclude that the main effect of font type was worth interpreting as, in general, people do prefer the serif font. The mathematics hasn't changed, but the context has – in this case the serif font seems like a sensible compromise that pleases most people most of the time (though even in this case the simple main effects might be useful for deciding what font to use for a motoring magazine aimed at men or a lifestyle magazine aimed at women). If there is a moral to be learned from this digression (other than that the author tends to ramble on a bit) it is that applied statistics is much more than mathematics and needs to take account of the context and the goals of the research.

4 In this case the terminology is a bit awkward and most authors would say "the simple main effect of age for the male participants" or similar. In other cases the terminology is quite handy.

5 This should be fairly obvious to work out if you think about it. With a 3x2 design there are 2 simple main effects for the first factor (one for each level of the other factor) plus 3 for the second factor (one for each level of the first factor).

6 Some readers will note that we could also calculate a t ratio if simple main effect has 1 d.f. for the treatment (as is the case for my example). This this t ratio would merely be the square root of the F ratio. In some cases the software one has available makes t easier or quicker to calculate than F so this relationship will prove useful.

7 Pooling the error term would also increase the potential for problems with the repeated measures sphericity assumption. If the simple main effect has 1 treatment d.f. then using a non-pooled error term will mean any problems with spheicity are avoided. If the simple main effect has more than 1 treatment d.f. and sphericity is considered to be untenable the researcher should probably consider whether the substantive hypothesis being investigated can be tested with a specific contrast of means. See also my guidance on checking and correcting sphericity violations.

8 In any case the terms to look for are the "Subjects within ..." term and the "x [or by] Subjects within ..." interaction term if they are not labelled as error terms. For example, a recent version of SPSS labels these simply "Error" in the "Tests of Between-Subjects Effects" table and "Error(Repeated factor A x Repeated factor B x ...)" in "Tests of Within-Subjects Effects".

What is all this stuff about sphericity in my repeated measures ANOVA output?

2006-05-22T16:54:00.004+01:00

Until relatively recently most psychologists had never heard of sphericity. Most text books didn't mention it (and even now some of them confuse it with compound symmetry). The big change happened a few years ago when SPSS started including information about sphericity in its repeated measures ANOVA output. Of course, because most undergarduate text books didn't mention it people started looking for information on it. Around that time I taught an advanced course on psychological statistics and produced this resource for my tutees. It turned out that they passed the web link on to other students on the course. Once I realized it was widely used I tried to keep it fairly up-to-date until I moved jobs and my web pages became homeless. I keep meaning to improve the layout and a little more detail in places, but in the mean time I hope a few people find it useful ...

An Introduction to Sphericity in Repeated Measures ANOVA

The article is written for a general audience of post-graduate and graduate researchers. The technical material goes slightly beyond what is covered in most text books, although there is still some simplification (which is usually indicated in the text). The aim to give advice about best practice for checking and dealing with sphericity in repeated measures ANOVA. Some of the content is personal opinion (which I have tried to indicate in the text). I include a short bibliography of my sources at the end for readers who want to explore the topic in more detail.

Some background

Sphericity is a mathematical assumption in repeated measures ANOVA designs. Let's start by considering a simpler ANOVA design (e.g., one-way independent measures ANOVA).

In independent measures ANOVA one of the mathematical assumptions is that the variances of the populations that groups are sampled from are equal. This homogeneity of variance assumption more-or-less follows from the null hypothesis being tested in ANOVA - if the treatment has no effect on the thing being measured (the DV) then we can consider all the groups to be sampled from the same population.¹However, because we're taking samples we'd be very lucky to observe exactly equal variances (even if the assumption were perfectly met). Real data is rarely that neat! What we'd expect to get (most of the time) is groups with similar variances.²

The sphericity assumption can be thought of as an extension of the homogeneity of variance assumption in independent measures ANOVA. Why does the assumption need to be extended? To understand this we need to introduce the ANOVA covariance matrix.

The covariance matrix

What is a covariance matrix? In a nutshell, it is a matrix that contains the covariances between levels of a factor in an ANOVA design.³ A covariance is the shared or overlapping variance between two things (sometimes called variance in common). Let's look at an example of the layout for a one-factor ANOVA design with four levels and therefore four samples (called A₁, A₂, A₃ and A₄):

Samples:	A₁	A₂	A₃	A₄
A₁	S₁²	S₁₂	S₁₃	S₁₄
A₂	S₂₁	S₂²	S₂₃	S₂₄
A₃	S₃₁	S₃₂	S₃²	S₃₄
A₄	S₄₁	S₄₂	S₄₃	S₄²

The first thing to notice is the main diagonal cells in the matrix (running top left to bottom right)contain the variances of the four levels (e.g., S₁² is the variance of A₁).⁴The second thing to notice is that the covariances are therefore in the cells off the main diagonals (called the off-diagonal cells and greyed out in the table above). The third thing to notice is that the covariances are mirrored above and below the main diagonal. (The term S₁₄ is the covariance between samples A₁ and A₄, while S₄₁is the covariance between samples A₄ and A₁. As this is the variance they have in common, S₁₄= S₄₁.)

What does the covariance matrix look like for independent measures ANOVA? Here an example for a one-way independent measures ANOVA design with 4 levels (and hence four groups):

Samples:	A₁	A₂	A₃	A₄
A₁	S₁²	0	0	0
A₂	0	S₂²	0	0
A₃	0	0	S₃²	0
A₄	0	0	0	S₄²

The most striking observation is that all the covariances are zero. Why? The answer is fairly straight-forward. In an independent measures design the observations should be independent and therefore uncorrelated with each other.⁵ Two samples that are uncorrelated will share no variance (and the covariance will be zero). So in this relatively simple case we only have to worry about homogeneity of variance - which would lead us to expect that the observed variances on the main diagonal should be similar.

Reminder: Assumptions such as homogeneity of variance, sphericity and so forth are assumptions about the populations we are sampling from. I'll try and indicate this as I go through, but it sometimes gets clumsy to keep repeating "in the population being sampled" all the time! We expect samples to have similar characteristics to the populations being sampled, but only in rare cases will the samples show exactly the same pattern of variance (or whatever) as the population. It is also worth adding that large samples are more similar to the populations they are sampled from than small samples.

Finally, please note that the statistical term "population" is an abstract one. We are referring to a population of data points that we might potentially be sampling not a fixed entity such as the population of a country. (In other contexts, such as market research, people sometimes deal with such fixed populations, but this requires slightly different methods from those in most scientific research).

What is the sphericity assumption?

Compound symmetry

The sphericity assumption is an assumption about the structure of the covariance matrix in a repeated measures design. Before we describe it in detail lets consider a simpler (but stricter) condition. This one is called compound symmetry. Compound symmetry is met if all the covariances (the off-diagonal elements of the covariance matrix) are equal and all the variances are equal in the populations being sampled. (Note that the variances don't have to equal the covariances.) Just as with the homogeneity of variance assumption we'd only rarely expect a real data set to meet compound symmetry exactly, but provided the observed covariances are roughly equal in our samples (and the variances are OK too) we can be pretty confident that compound symmetry is not violated.

The good news about compound symmetry

If compound symmetry is met then sphericity is also met. So if you take a look at the covariance matrix and the covariances are similar and the variances are similar then we know that sphericity is not going to be a problem.⁶

The bad news about compound symmetry

As compound symmetry is a stricter requirement than sphericity we still need to check sphericity if compound symmetry isn't met. This is where it gets technical (or should that be ... even more technical).

The sphericity assumption

Lets take a look at some raw data. Imagine that the first few observations of A₁, A₂, A₃ and A₄are as follows:

	A₁	A₂	A₃	A₄
_{Participant 1}	8	9	12	4
_{Participant 2}	6	11	16	3
_{Participant 3}	9	8	12	5
_etc.	...	...	...	...

For each possible pair of levels of factor A (e.g., A₁ and A₂or A₂ and A₃) we can calculate the difference between the observations. For example:

	A₁- A₂	A₁- A₃	A₁- A₄	etc.
_{Participant 1}	-1	-4	+4	...
_{Participant 2}	-5	-10	+3	...
_{Participant 3}	+1	-3	+4	...
_etc.	...	...	...	...

We could then calculate variances for each of these differences (e.g., S_1-4²). The sphericity assumption is that the all the variances of the differences are equal (in the population sampled). In practice, we'd expect the observed sample variances of the differences to be similar if the sphericity assumption was met.

Using the covariance matrix to check the sphericity assumption

We can check sphericity assumption using the covariance matrix, but it turns out to be fairly laborious. (Later on I'll discuss some simpler ways to check sphericity using output from SPSS and similar statistics packages). Variance of differences can be computed using a version of the variance sum law:

S_x-y² = s_x² + s_y² - 2(s_xy)

In other words the variance of a difference is the sum of the two variances minus twice their covariance. A simple arithmetic check will show that this works out as zero if the two variances share all their variance.

(Note that we could also calculate the variances of the differences directly from the raw data. We'd simply calculate the differences between all the possible pairs of levels of a factor. For example, using Excel or SPSS we could define a new column value as one level minus another level and then calculate the variances of each column using the built in descriptive statistics of the program. This would get very tedious if we had lots of levels, so I'd recommend the above method if you really want to calculate the variances of the differences and you already have a covariance matrix. Fortunately this isn't necessary in most cases - as I'll discuss later.)

An example

This example is adapted from Kirk (1995). Imagine the observed covariance matrix for a one-way repeated measures ANOVA design is this:

Samples:	A₁	A₂	A₃	A₄
A₁	10	5	10	15
A₂	5	20	15	20
A₃	10	15	30	25
A₄	15	20	25	40

The variances of the differences are:

S_x-y² = S_x² + S_y² - 2(S_xy)

So ...

S_1-2² = S₁² + S₂² - 2(S₁₂) = 10 + 20 - 2(5) = 30 - 10 = 20

S_1-3² = 10 + 30 - 2(10) = 20

S_1-4² = 10 + 40 - 2(15) = 20

S_2-3² = 20 + 30 - 2(15) = 20

S_2-4² = 20 + 40 - 2(20) = 20

S_3-4² = 30 + 40 - 2(25) = 20

This example has been contrived so that variances of the differences are exactly equal (which would be extremely unlikely in real data), but it does demonstrate that lack of compound symmetry does not necessarily mean that sphericity is violated. (Compound symmetry is a sufficient, but not necessary requirement for sphericity to be met.)⁷ In this example, compound symmetry is clearly not met (the largest variances and covariances are 4 or 5 times bigger than the smallest), but sphericity holds.

What to do if sphericity is violated in repeated measures ANOVA

There are two broad approaches to dealing with violations of sphericity. The first is to use a correction to the standard ANOVA tests. The second is to use a different test (i.e., one that doesn't assume sphericity).

In the following sub-sections I give general advice on what to do if sphericity is violated, this advice tends to hold well in most cases for factorial repeated measures designs but may be problematic for mixed ANOVA designs (discussed later under Complications).

Correcting for violations of sphericity

The best known corrections are those developed by Greenhouse and Geisser (the Greenhouse-Geisser correction) and Huynh and Feldt (the Huynh-Feldt correction).⁸ Each of these corrections works roughly in the same way. They all attempt to adjust the degrees of freedom in the ANOVA test in order to produce a more accurate significance (p) value. If sphericity is violated the p values
need to be adjusted upwards (and this can be accomplished by adjusting the degrees of freedom downwards).

The first step in each test is to estimate something called epsilon.⁹For our purposes we can consider epsilon to be a descriptive statistic indicating the degree to which sphericity has been violated. If sphericity is met perfectly then epsilon will be exactly 1. If epsilon is below 1 then
sphericity is violated. The further epsilon gets away from 1 the worse the violation.

How bad can epsilon get? Well, it depends on the number of levels (k) on the repeated measures factor:

Lower bound of epsilon = 1/(k-1)

Therefore, epsilon can go as low as 1/(3-2) = 0.5 for three levels, as low as 0.2 for six levels and so forth. The more levels on the repeated measures factor the worse the potential for violations of sphericity.¹⁰

The three common corrections fall into a range of most to least strict. First consider the most strict. We could use the lower bound value of epsilon and correct for the worst possible case. Fortunately, there is a much better option. The Greenhouse-Geisser correction is a conservative correction (it tends to underestimate epsilon when epsilon is close to 1 and therefore tends to over-correct). Huynh-Feldt produced a modified version for use when the true value of epsilon is thought to be near or above 0.75.

The Huynh-Feldt correction tends to overestimate sphericity, so some statisticians have suggested using the average of the Greenhouse-Geisser and Huynh-Feldt corrections. My advice would be to consider the aims of the research and the relative cost of Type I and II errors. If Type I errors are considered more costly (especially if the estimates of epsilon fall below 0.75) then stick to the more conservative Greenhouse-Geisser correction.

Using the correction is fairly simple. Replace the treatment and error d.f. by (epsilon*d.f.). So an epsilon value of 0.6 would turn an F_1,50 test into an F_0.6,30 test.¹¹Using the these corrections seems to work well for relatively modest departures from 1 by epsilon or when sample sizes are small.

Using MANOVA

An alternative approach is to use a test that doesn't assume sphericity. In the case of repeated measures ANOVA this usually means switching to multivariate ANOVA (MANOVA for short). Some computer programs print out MANOVA automatically alongside repeated measures ANOVA (SPSS is one of these). While this can be confusing, it does make it easy to compare results for different tests and corrections. If sphericity is met (i.e., epsilon = 1) all the p values for a given test should be identical. The degree to which they differ can be informative. If there is a wide discrepancy between different tests or correction then this suggests that the sphericity assumption may be severely violated and that one of the more conservative tests should be reported (e.g., Greenhouse-Geisser corrected F or MANOVA).

In general MANOVA is probably less powerful than repeated measures ANOVA and therefore should probably be avoided. However, when some experts suggest sample sizes are reasonably large (n exceeds 10 + k) and epsilon is low (less than 0.7) MANOVA will typically be more powerful and should probably be preferred. The most up-to-date evidence suggests that the precise circumstances under which MANOVA is more powerful depend on the properties of the covariance matrix and k in ways that are hard to predict.

My view is that most psychological research should avoid MANOVA and prefer sphericity assumed or corrections based on Huynh-Feldt or Greenhouse-Geisser epsilon estimates (especially when multisample sphericity - see below - is an issue). This assumes two things. First, that the important tests can not be reduced to a 1 effect d.f. contrasts with a non-pooled error term (see Multiple comparisons and contrasts below). Second, that psychologists tend to deal with relatively small k and small N situations. I don't think that there is much evidence that where MANOVA is more powerful that it is much more powerful than ANOVA for the modest values of n or k that appear in most of the psychological literature. I suspect that greater gains in power would be achieved by looking at other aspects of the study in most cases (e.g., see Baguley, 2004). If high quality pilot data were available it would be possible to estimate the the power of MANOVA or ANOVA in this situation (e.g., see Miles, 2003). However, I suspect that a small pilot sample would probably not estimate the population covariance with sufficient precision to be confident in this approach (this is not a flaw in the approach, but merely the observation that there is error in the estimates of all parameters in a sample - not just for the mean).

How to check sphericity

In this section I will focus on information readily available in SPSS (and most good statistics packages).

Factorial repeated measures ANOVA

If there is more than one repeated measures factor consider each factor separately. (See also Special cases: factors with 2 levels.)

A warning about Mauchly's sphericity test

Many text books recommend using significance tests such as Mauchly's to test sphericity. In general this is a very bad idea.

Why? First, tests of statistical assumptions - and Mauchly's is no exception - tend to lack statistical power (they tend to be bad at spotting violations of assumptions when N is small). Second, tests of statistical assumptions - and, again, Mauchly's is no exception - tend not to be very robust (unlike ANOVA and MANOVA they are poor at coping with violations of assumptions such as normality). Third, significance tests don't reveal the degree of violation (e.g., with very large N even a poor test like Mauchly's will show significance if there are very minor violations of sphericity; with low N the poor power means that even severe violations may not be detected). Fourth, significance tests of assumptions tend to be used as substitutes for looking at the data - if you followed the advice of many popular introductory texts you'd never look at the descriptive statistics at all (e.g., the variances, the covariance matrix, estimates of epsilon and so forth). Fifth, I just don't like them.¹²

Using Mauchly's sphericity test

The test principle is fairly simple. The null hypothesis is that sphericity holds (I like to think of it as a test that the true value of epsilon = 1). A significant result indicates evidence that sphericity is violated (i.e., evidence that the true value of episilon is below 1).

Epsilon

I would recommend using estimates of epsilon to decide whether sphericity is violated. If epsilon is close to 1 then it is likely that sphericity is intact (or that any violation is very minor). If epsilon is close to the lower bound (see Correcting for violations of sphericity above) then a correction or alternative procedure is likely to be necessary. Exactly where to draw the line is a matter of personal judgement, but it is often instructive to compare p values for the corrected and uncorrected tests. If they are fairly similar then there is little indication that sphericity is violated. If the the discrepancy is large then one of the corrections (or possibly MANOVA) should probably be used.

The covariance matrix

If estimates of epsilon are not readily available then lower-bound procedures can be used (see above) or the covariance matrix can be consulted. If compound symmetry holds then it is safe to proceed with repeated measures ANOVA. If compound symmetry does not hold it is relatively simple (if time-consuming) to calculate the variances of the differences for each factor from the covariance matrix.

Complications

Special cases: factors with 2 levels (and the paired t test)

If k = 2 (a repeated measures factor with only two levels) then the sphericity assumption is always met. Using the lower-bound formula one can see that when k = 2 epsilon can't be lower than 1/(k-1) = 1/(2-1) = 1. This is also true for the paired t test (in effect a one-way repeated measures ANOVA where k = 2).

Why isn't sphericity a problem when there are only two levels? Well, think about the covariance matrix:

Samples:

A₁

A₂

A₁

S₁²

S₁₂

A₂

S₂₁

S₂²

There are two covariances s₂₁ and s₁₂. The covariances above and below the main diagonal are constrained to be equal (because the shared variance between level 1 and level 2 is the same thing as the shared variance between level 2 and level 1). In effect there is only one covariance. Similarly, if we calculated s_1-2²(the variance of the difference) we should realize there is, in effect, only one such variance. Sphericity is met if all the variances of the differences are equal. As there is only one, it can't not be equal to itself. For this reason Mauchly's sphericity test can't be computed if d.f. = 1 (i.e, if k = 2) and some computer programs give confusing messages or printouts if you try. This sometimes leads people to conclude that sphericity is violated when sucah a violation would be impossible.

Note that sphericity subsumes the standard homogeneity of variance assumption. In effect, we are only interested in the variances of the differences. When k = 2 there is only one variance of the difference between levels and we can ignore differences in the 'raw' level variances themselves.

Multiple comparisons and contrasts

Bonferroni t tests are frequently recommended for repeated measures ANOVA (whether or not sphericity is violated). The Bonferroni correction relies on a general probability inequality and therefore isn't dependent on specific ANOVA assumptions. As Bonferroni corrections tend to be conservative, a number of modified Bonferroni procedures have been proposed. Some are specific to certain patterns of hypothesis-testing, but others such as Holm's test (or the equivalent Larzelere and Mulaik test) are more powerful than standard Bonferroni corrections and should be used more widely (and not just for ANOVA). The superficially similar Hochberg procedure is usually much more powerful than Bonferroni and modified Bonferroni procedures and might also be considered. The Hochberg procedure is radically different in philosophy from standard multiple comparison procedures (though similar to the Holm procedure in its computation) and should be considered carefully before adopted).

Experts recommend specific (rather than pooled) error terms for repeated measures factors (i.e., calculate the SE for t using only the conditions being compared, rather than using the square root of the MSE term from the ANOVA table to derive the pooled SD and hence compute a pooled SE).

This advice also extends to contrasts which can be easily calculated by performing paired t tests on weighted averages of the appropriate means. Using a specific error term should avoid problems with sphericity (e.g., see Judd et al., 1995) for the same reason that sphericity is not a problem for factors with only 2 levels.

In complex ANOVA or MANOVA designs an approach involving a priori contrasts is also preferable for other reasons. Such designs leads to multiple tests of theoretically uninteresting hypotheses whereas selecting a small number of contrasts on the basis of prior theory is to be preferred on the basis of clarity, statistical robustness, control of Type I error and statistical power.

Mixed designs

Mixed designs (combining independent and repeated measures factors) muddy the waters somewhat. Mixed measures ANOVA requires that multisample sphericity holds. This more-or-less means that the covariance matrices should be similar between groups (i.e., across the levels of the independent measures factors). Provided group sizes are equal (or at least roughly equal) the Greenhouse-Geisser and Huynh-Feldt corrections perform well when multisample sphericity doesn't hold and can therefore still be used. If these corrections are inappropriate, or if group sizes are markedly unequal then more sophisticated methods are required (Keselman, Algina & Kowalchuk, 2001). A description of these methods is beyond the scope of this summary (possible solutions include multilevel methods found in SAS PROC MIXED, MlWin and HLM, though Keselman et al. also discuss a number of other options). For most psychological research best advice is, if at all possible, to keep group sizes in mixed ANOVA equal (or as close to equal as feasible).

Bibliography

Baguley, T. (2004). Understanding statistical power in the context of applied research. Applied Ergonomics, 35, 73-80.

Field, A. (1998). A bluffer's guide to ... sphericity. The British Psychological Society: Mathematical, Statistical & Computing Section Newsletter, 6, 13-22.

Howell, D. C. (2002). Statistical Methods for Psychology. (5th. ed.). Belmont, CA: Duxberry Press.

[David Howell has just released a sixth edition which looks excellent. At first glance the main changes are in layout and use of software examples. Once I'm sure of the the content I'll update this reference.]

Judd, C. M., McClelland, G. H., & Culhane, S. E. (1995). Data analysis: continuing issues in everday analysis of psychological data. Annual Review of Psychology, 46, 433-465.

Keselman, H. J., Algina, J., & Kowalchuk, R. K. (2001). The analysis of repeated measures designs: a review. British Journal of Mathematical and Statistical Psychology, 54, 1-20.

Kirk, R. E. (1995). Experimental Design: Procedures for the Behavioral Sciences. (3rd ed.). Pacific Grove: Brooks/Cole.

Miles, J. (2003). A framework for power analysis using a structural equation modelling procedure. BMC Medical Research Methodology, 3: 27. Published online 2003 December 11. doi: 10.1186/1471-2288-3-27.

Footnotes:

¹ As long as the treatment only has the effect of adding or subtracting to the group means (and doesn't influence their variances) the homogeneity of variance assumption isn't a problem. This special case is known as unit-treatment-additivity. Unfortunately life isn't always that simple: there are good reasons why treatments might be expected to influence both means an variances. For this reason it is always sensible to check the group variances in independent measures designs.

² As a rule of thumb the largest group variance should be no more than three or four times as large as the smallest group variance.

³ Covariance matrices also crop up in all sorts of other statistics, but we can forget about that for now.

⁴The diagonals contain the variances because samples share all of their variance with themselves. I've used s rather than the Greek sigma symbol because it turns out better when browsers with different fonts are used. s is normally used for samples and sigma for populations, but I'm using s throughout for convenience (sorry!). You can generate Greek letters by using the "symbol" font in many word processors (e.g., 's' for sigma, m for mu and so forth).

⁵The covariance between groups will rarely be exactly zero in the samples. However, provided people are randomly assigned to groups, and each person contributes only one data point then we can be pretty certain that the covariances in the populations being sampled are zero (and therefore the independence assumption is met). Even if random assignment to groups doesn't occur the independence assumption is often reasonable. Any time we know or believe that the measures will be correlated (e.g., in matched designs) a repeated measures analysis should be used. (I use repeated measures in a non-technical rather than just to mean a type of ANOVA or general linear model).

⁶I probably should have mentioned this earlier, but covariances (like correlations) can be both negative and positive (unlike variances which are always positive). Positive covariances occur between samples when two samples are positively correlated. Negative covariances occur between samples when two samples are negatively correlated. The idea of a negative covariance is often tricky to grasp - but it just means that as one group tends to vary upwards in value the other tends to vary downwards. So when checking covariances to see if they are similar bear in mind the sign of the covariance as well as its magnitude (e.g., a covariance of -124.3 is very different from +124.3).

⁷By now you can probably appreciate why many text books focus on compound symmetry and don't cover sphericity in detail.

⁸One of the nice things about this topic is that the tests have nice, impressive-sounding names.

⁹The Greek letter epsilon is usually used. Greenhouse-Geisser estimates of epsilon have a little hat on top (^). Huynh-Feldt estimates have a little squiggle on top (~). You can generate Greek letters by using the "symbol" font in many word processors (e.g., 'e' for epsilon).

¹⁰Later on we discuss the special case of k = 2 and the analagous case of paired t tests. Feel free to jump ahead if you wish.

¹¹You won't find tables for fractional d.f., but exact p values can be calculated if d.f. are fractional (most good computer packages do this automatically these days).

¹²Why don't I like them? Apart from all the above reasons, I don't like the idea of using a significance test to test the assumptions of a significance test. If that was a good idea, why don't we use significance tests to test the assumptions of Mauchly's sphericity test or Levine's test of homogeneity of variances? At some point you've got look at the data (using graphical methods, descriptive statistics and so forth) and make a considered judgement about what procedures to use. My view is that the earlier this is done in the process the better ...

What is a psychological statistics blog?

2006-05-10T13:03:00.001+01:00

I've no idea.

Some time ago I had the idea of a blog about the kinds of statistics I use in my work. I didn't really think much about it until I noticed a couple of blogs by a friend of mine (Learning Statistics and Applying Regression by Jeremy Miles). Jeremy's blogs pointed me to the excellent Statistical Modeling, Causal Inference and Social Science by Andrew Gelman and Samantha Cook. As a consequence I'm not sure if this means that a psychological statistics blog is where all the cool academics hang out (though ... on reflection it seems unlikely) or whether I'm just slow to catch on (as usual).

The reason the blog is appearing now is really just to make a home for all sorts of odds and ends I've written or (half-written) that have never seen the light or day, or were published but have since become homeless. What this blog is about might never become clear, but then (as a newcomer to blogging) I suspect if I did have a clear idea what I was doing I'd write a book instead.