Psychological Statistics

Neuroscience, statistical power and how to increase it

2013-04-21T20:50:00.000+01:00

There has been quite a bit of buzz recently about the Button et al. Nature Reviews Neuroscience paper on statistical power. Several similar reviews have been published in psychology and other disciplines and come to broadly the same conclusion - that most studies are underpowered. The main difference with the Button et al. study is that they don't just find that typical studies are underpowered to detect the average size of effect in a field, but they find extremely low power in neuroscience research (around 20%, and below 10% for some subfields). Contrast this with a typical review from psychology and related disciplines. Sedlmeier and Gigerenzer (1989, Table) report power to detect a medium effect size ranging from 37% to 89%. David Clark-Carter (1997) reviewed papers in the British Journal of Psychology and found power to detect a median effect of 59%. Thus the power of typical research in psychology is not that high, but (if we make fairly reasonable assumptions about the size of typical effects in the discipline) estimates appear to be around 60% rather than the 20% found in the Button et al. paper. What caught me interest, however, was some of the responses to the publication in blogs and blog comments. For example one of the comments on Ed Yong's piece stated

Another argument for parallel recording. Traditional, one-neuron-at-a-time neurophysiological papers study 10s of neurons. Multi-electrode studies have 100s or 1000s of neurons. Enough power? Maybe not, but way more power than single neuron recording.

A similar sentiment arises in Matt Wall's piece:

MRI scanners have significantly improved in the last ten years, with 32 or even 64-channel head-coils becoming common, faster gradient switching, shorter TRs, higher field strength, and better field/data stability all meaning that the signal-to-noise has improved considerably. This serves to cut down one source of noise in fMRI data – intra-subject variance. The inter-subject variance of course remains the same as it always was, but that’s something that can’t really be mitigated against, and may even be of interest in some (between-group) studies. On the analysis side, new multivariate methods are much more sensitive to detecting differences than the standard mass-univariate approach.

Matt's piece is thoughtful and I would agree which much of what he writes, but the idea that increasing observations within a person will do much to resolve the problem is probably not correct (and for reasons that Matt mentions). To understand why, consider the typical nature of the experimental designs being used. As I understand it there are essentially two main types of design: a nested repeated measures design or a factorial design with fully crossed random effects. There are many variants (e.g., additional layers of nesting, additional fully crossed random factors), but the aforementioned characteristics capture characteristics most of the designs I'm familiar with in cognitive neuroscience (and possibly in many other areas of neuroscience).

In a nested repeated measures design there are m multiple measurements within each of n persons. The multiple measurement are correlated in some way so - in general - the power of the design has an effective sample size that is less than N (where N = n * m). It turns out that for most such designs the limiting factor in power and precision is n and not m or N.

This isn't always true, but generally experimental designs get refined quite quickly to reduce the impact of sources of error in the repeated measurements. This could be increasing the number of trials or tightening up the experimental procedures (e.g., instructions, quality of materials) or by technical advances that reduce measurement error for each measurement occasion. Once you get measurement error per trial moderately low, improving measurement error further has very little impact on power. That's because the error at each measurement occasion includes transient error that can't really be eliminated (many behaviours are just inherently variable from occasion to occasion) and because as you reduce these errors the other sources of error in the study become the main limiting factors on power.

For example, when I was a PhD student many reaction time experiments used computers with dodgy clocks that couldn't time more accurately than 1/60th of a second or around 17 ms (and perhaps many still do). If you are looking for a priming effect of say 30 milliseconds this would seem like a major problem. However, you can get pretty accurate inferences without much bias or loss of power as long as the variability of the RTs are fairly large - which they generally are (Ulrich & Giray, 1989). For most neuroscience work involving humans the limiting factors in power (once you are dealing with a reasonably refined experimental set-up) are therefore related to n. A further consideration is that top level n generally needs to be in the 30-50 range or (preferably) greater just to get vaguely reasonable estimates of the variances and covariances if you are dealing with data sampled from approximately normal distributions. Smaller samples also make the study more vulnerably to a atypical 'outlier' at the person level (e.g., a participant using a weird strategy or responding randomly) or to selective bias by the experimenters (dropping a 'noisy' participant because they go against the hypothesis). Having small n at the top level may also make focus on statistical significance rather than interval estimates of effects more attractive (because it reduces precision of measurement). In other words it encourages studies that find 'evidence' of an effect and discourages focus on accurate estimates of the size of an effect.

For fully crossed random factor designs the situation is worse. In these designs you sample both people and stimuli (e.g., faces, words, etc.) from a large (conservatively assumed to be infinite) population. The limiting factor on power now probably depends not on n1 (the number of people) or n2 (the number of stimuli) but the the smaller of n1 and n2 (assuming you want to make inferences that generalise to people and stimuli not in your experiment). Thus having 1000 people has little effect on power if your study uses only two faces (and you want to make general inferences about face perception rather than perception of those two faces). This is a slight oversimplification - as it assumes that the stimuli and people are equally variable in terms of what you measure - however it is a good rule of thumb unless variability in either people or stimuli is large enough to swamp the other source.

There is also an important caveat here - I'm assuming that you do the statistics correctly. Many, many studies still analyse fully crossed random factor designs as if they are nested, resulting in spuriously high power (see here for an earlier blog post on this).

This analysis should hold whenever: i) the basic experimental procedure is fairly well-refined, ii) variability between people (or stimuli in appropriate designs) on the measures of interest are non-negligible. Thus it should hold more often than not in psychology and related areas of neuroscience. There are undoubtedly subfields in which it won't hold (e.g., some areas of vision research where n = 2 studies are common because individual differences on the crucial effects are low).

Postscript

One objection to my conclusion is that if neuroscience power is limited by number of participants and number of stimuli, why do small samples persist? This is a good question. I offer three main answers: i) As with psychology (where power is also generally low, remember) you can have low power for each test if you have multiple tests. Maxwell (2004) pointed out that a typical 2 x 2 factorial design might only have 50% power per test but that means 87.5% chance of at least one significant result. Thus low power generally produces something statistically significant (though it also predicts that replications will generally fail to show consistent patterns of statistical significance), ii) researcher degrees of freedom (see Simmons et al., 2011), and iii) many research teams run many small studies (e.g., undergraduate and masters projects) so (in some cases) there are many unreported studies with null results.

References

Maxwell, S. E. (2004). The persistence of underpowered studies in psychological research: causes, consequences, and remedies. Psychological Methods, 9, 147–63.

Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011). False-positive psychology: undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological Science, 22, 1359-66.

Ulrich, R., & Giray, M. (1989). Time resolution of clocks: Effects on reaction time measurement - Good news for bad clocks. British Journal of Mathematical & Statistical Psychology, 42, 1-12.

Reflecting on the end of history illusion illusion

2013-04-10T21:38:00.002+01:00

A while back Jon Sutton at The Psychologist asked my opinion on the end of history illusion. This was sparked by an interesting Science paper by Quoidbach, Gilbert and Wilson. Blogger and mathematician Jordan Ellenberg had written a blog post arguing that the paper makes a mistake: "a somewhat subtle mistake, but a bad mistake, and one which kills a big chunk of the paper".

Jon wanted a second opinion, and after a bit of reading I replied that Ellenberg's criticisms were valid. I meant to blog about it at the time but got caught up in other things. Consequently I missed the BPS research digest piece on it.

The reason for writing this blog post is because the flaw that Ellenberg spotted is quite interesting in its own right and because both the description by Ellenberg and the description in the Research Digest article probably don't explain it clearly enough for some readers to appreciate. Ellenberg's piece is (I hasten to add) crystal clear but relies on a reader being comfortable with the formal, mathematical approach he takes (which many psychologists won't be). The Research Digest description just gives the brief gist (with a link to Ellenberg for the full picture). Here is my belated attempt at a psychologist-friendly interpretation with no formal notation - and as little maths as possible.

According to the end of history illusion people underestimate how much they will change in the future. For example, someone asked to predict how their personality would change in the next ten years would come up with a prediction closer to their original position than their actual position. Quoidbach et al. tested this mainly by asking people to predict future values on some psychological variable (e.g., a personality test score) and then showing that actual change is much greater than the difference between the original and predicted scores. This seems highly plausible, but Ellenberg pointed out that the difference in the predicted and original scores is a different quantity from the expected (absolute) change in scores.

Why is this? Perhaps the easiest way to understand is to work through a simple example. Imagine that my extraversion score is 50 on a scale that goes from 0 (extremely introverted) to 100 (extremely extraverted). A researcher then asks me to predict my extraversion score in 10 years time. I, being a keen observer of human nature (bear with me on this if you know me - it is just an example), am aware that personality is not fixed and judge that I am likely to change quite a bit - say 15 points - on the scale. However, I might get more extraverted or I might get more introverted (depending on how life treats me over the next ten years). Given that I'm in the middle of the scale, I could end with a score of 35 or a score of 65. Thus I predict that my extraversion score after 10 years will be (35 + 65)/2 = 50. It looks as though I've predicted zero change, when what I've done is give the best prediction I can (one that minimizes my prediction error). Had I instead been asked to give the absolute change I expected, my answer would have been different. It would have been (15 + 15)/2 = 15 (not zero).

Although the example is simple it captures the essence of the problem. Commenters on Ellenberg's blog looked again at the raw data that Quoidback et al. provided. According to their analyses the end of history illusion largely disappears when analyzed correctly (though only some of the data sets support such a reanalysis). Thus if the end of history illusion effect exists (and the basic premise seems highly plausible) it is quite probably a much smaller and more fragile effect than originally thought. That makes sense to me - because I'm not sure that such a bias could be both pervasive and large in the face of the counter-evidence available to people about past change in themselves and change in others.

My continued interest in the effect is slightly different. There seems to be a cognitive illusion at work here - one that makes the difference between the original score and predicted score appear to be a good measure of an entirely different quantity - the expected absolute change in score ...

The growth of Bayesian methods in psychology

2013-01-28T21:27:00.001Z

The British Journal of Mathematical and Statistical Psychology has published a target article (with commentaries and reply) by Andrew Gelman and Cosma Shalizi on philosophy and the practice of Bayesian statistics.

Mark Andrews and I introduce the target article with an editorial aimed at providing some background to psychologists who are interested in Bayesian statistics but need a little back story. Our main aim was to try and indicate that the debate about Bayesian statistics has moved on from the frequentist vs. Bayesian argument and on to more interesting territory - illustrated both by the target article and the commentaries.

Also I believe that as of writing access is free to the target article and commentary ...

Andrews, M., & Baguley, T. (2013). Prior approval: The growth of Bayesian methods in psychology. British Journal of Mathematical and Statistical Psychology, 66, 1–7. doi:10.1111/bmsp.12004

Gelman, A., & Shalizi, C. R. (2013). Philosophy and the practice of Bayesian statistics. British Journal of Mathematical and Statistical Psychology, 66, 8–38. doi:10.1111/j.2044-8317.2011.02037.x

Guest post: Visualizing data using a 3D printer

2012-09-25T23:10:00.001+01:00

In a break from my usual obsessions and interests here is a guest blog post by Ian Walker. I'm posting it because I think it is rather cool and hope it will be of interest to some of my regular readers. Ian is perhaps best known (in the blogosphere) for his work on transport psychology - particularly cycling - but is also an expert on psychological statistics.

Some time ago, I had some data that lent themselves to a three-dimensional surface plot. The problem was, the plot was quite asymmetrical, and finding the right viewing angle to see it effectively on a computer screen was extremely difficult. I spent ages tweaking angles and every possible view seemed to involve an unacceptable compromise.

Of course, displaying fundamentally three-dimensional items in two dimensions is an ancient problem, as any cartographer will tell you. That night, as I lay thinking in bed, a solution presented itself. I had recently been reading about the work of a fellow University of Bath researcher, Adrian Bowyer, and his RepRap project, to produce an open-source three-dimensional printer. The solution was obvious: I had to find a way to print R data on one of these printers!

I managed to meet up with Adrian back in May 2012, and he explained to me the structure of the STL (stereolithography) files commonly used for three-dimensional printing. These describe an object as a large series of triangles. I decided I'd have a go at writing R code to produce valid STL files.

I'm normally a terrible hacker when it comes to programming; I usually storm in and try to make things work as quickly as possible then fix all the mistakes later. This time, I was much more methodical. As a little lesson to us all, the methodical approach worked: I had the core code producing valid STL files in under 3 hours.

Unfortunately, it then took until September 2012 before I could get hold of somebody with a 3D printer who'd let me test my code. A few days ago the first prototype was produced, as you can see in this photograph:

So now I'd like to share the code under a Creative Commons BY-NC-SA licence, in case anybody else finds it useful. You can download the code here, in a file called r2stl.r. One day, when I learn how, I might try to make this a library, but for now you can just call this code with R's source() command. All that is in the file is the function r2stl(), and having once called the file with source(), you can then use the r2stl function to generate your STL files. The command is:

r2stl(x, y, z, filename='3d-R-object.stl', object.name='r2stl-object', z.expand=FALSE, min.height=0.008, show.persp=FALSE, strict.stl=FALSE)

x, y and z should be vectors of numbers, exactly as with R's normal persp() plot. x and y represent a flat grid and z represents heights above this grid

filename is pretty obvious, I hope

object.name The STL file format requires the object that is being described to have a name specified inside the file. It's unlikely anybody will ever see this, so there's probably no point changing it from the default

z.expand By default, r2stl() normalizes each axis so it runs from 0 to 1 (this is an attempt to give you an object that is agnostic with regard to how large it will eventually be printed). Normally, the code then rescales the z axis back down so its proportions relative to x and y are what they were originally. If, for some reason, you want your 3D plot to touch all six faces of the imaginary cube that surrounds it, set this parameter to TRUE

min.height Your printed model would fall apart if some parts of it had z values of zero, as this would mean zero material is laid down in those parts of the plot. This parameter therefore provides a minimum height for the printed material. The default of 0.008 ensures that, when printed, no part of your object is thinner than around 0.5 mm, assuming that it is printed inside a 60 mm x 60 mm x 60 mm cube. Recall that the z axis gets scaled from 0 to 1. If you are printing a 60mm-tall object then a z-value of 1 represents 60mm. The formula is min.height=min.mm/overall.mm, so if we want a minimum printed thickness of 0.5mm and the overall height of your object will be 60mm, 0.5/60 = 0.008, which is the default. If you want the same minimum printed thickness of 0.5mm but want to print your object to 100mm, this parameter would be set to 0.5/100 = 0.005

show.persp Do you want to see a persp() plot of this object on your screen as the STL is being generated? Default is FALSE

strict.stl To make files smaller, this code cheats and simply describes the entire rectangular base of your object as two huge triangles. This seems to work fine for printing, but isn't strictly proper STL format. Set this to TRUE if you want the base of your object described as a large number of triangles and don't mind larger files

To view and test your STL files before you print them, you can use various programs. I have had good experiences with the free, open-source Meshlab, which even has iPhone and Android versions so you can let people interact with your data even when you're in the pub. Even if all you ever do is show people your 3D plots using Meshlab, I believe r2stl() still offers a useful service, as it makes viewing data far more interactive than static persp() plots. To actually get your hands on a printer, you might try your local school - apparently lots of schools have got rapid prototypers these days.

Demo


source('r2stl.r')

# Let's do the classic persp() demo plot, as shown in the photograph above

x <- seq(-10, 10, length= 100)

y <- x

f <- function(x,y) { r <- sqrt(x^2+y^2); 10 * sin(r)/r }

z <- outer(x, y, f)

z[is.na(z)] <- 1

r2stl(x, y, z, filename="lovelyfunction.stl", show.persp=TRUE)



# Now let's look at R's Volcano data

z <- volcano

x <- 1:dim(volcano)[1]

y <- 1:dim(volcano)[2]

r2stl(x, y, z, filename="volcano.stl", show.persp=TRUE)

I hope you might find this code useful. Any questions or suggestions, then please get in touch.

September 2012 - Ian Walker, Department of Psychology, University of Bath.

Yet more on p values ...

2012-08-21T21:29:00.003+01:00

I wasn't going to post on this ... but couldn't resist. A recent QJEP paper reports suspicious patterns in p values across three psychology journals.

This has been blogged elsewhere (see here and here), so I haven't got too much to add. Although I generally like the paper and am glad it got published in a decent journal (I'm an EPS member and subscriber so I'm glad they published it), I can't say I find the main finding surprising. Everything we know about how significance testing is used in practice would predict the basic pattern of a bump just below p = .05.

My main criticism of the paper would be that they didn't use a multilevel model to account for dependencies in p values from the same paper - while I agree with the authors that the outcome wouldn't change, I think it could be potentially informative to model the dependency (and a simple multilevel model shouldn't be more than a few minutes to set-up and run if the data are coded by study). An even better approach for future work might be to sample papers from multiple journals and thus estimate the stability of the pattern across journals, disciplines etc.

The main reason I'm blogging is because of a few points raised at other blogs. The generally excellent BPS research digest reported:

Masicampo and Lalande said their findings pointed to the need to educate researchers about the proper interpretation of null hypothesis significance testing and the value of alternative approaches, such as reporting effect sizes and confidence intervals. " ... [T]he field may benefit from practices aimed at counteracting the single-minded drive toward achieving statistical significance," they said.

I couldn't find the effect sizes comment in the paper (but it may have been from conversations with the authors). I'm pretty sure all the journals surveyed reported effect sizes - point estimates of the magnitude of the effect. Perhaps they meant standardized effect size, but if so the advice is in my view doubly wrong. Standardized effect size has all sort of problems (see here and here for my take).

I won't repeat myself too much over this, but point out that standardized effect size generally obscures the thing you want to measure (as well as distorting it in several ways). That is the first reason why they aren't a solution to dodgy analyses - they will tend to make it harder for people to spot the problems in a study when there are meaningful, well-understood units (such as percentage recall, response times and so forth). They can maybe, sometimes help you interpret arbitrary measures - but generally only if you don't know the range of the scale. For instance, a shift of 3 points on a 7 point scale is huge and might make people suspicious whereas an effect size estimate of d = .6 or r = .4 might not.

The second reason is that effect size in general (and standardized effect size in particular) is at least as vulnerable as p values to publication bias, optional stopping and all the other known problems with statistical significance as routinely practiced. Combine small studies with a threshold for publication (e.g., p < .05 or d > 0.5) and effect size is substantially biased upwards (the bias is lower for large studies). Standardized effect size makes this worse because small samples tend to underestimate population variance and thus inflate the effect size estimate yet further. The final straw is that standardized effect size is easy to game. For instance, Cohen's d type metrics for a paired design come in at least four flavours (removing between-subject variance or not, crossed with a pooled or non-pooled error standard deviation). Removing the between-subject effect increases the reported d value artificially relative to an independent subjects design (even though the magnitude of the effect is generally assumed not to change). A similar problem occurs with variance explained measures such as eta-squared. People generally report partial eta-squared inappropriately in preference to classical eta-squared (which is generally smaller except in very simple designs).*

So, effect size is a red herring. The point on confidence intervals (and interval estimates in general) is a good one.

The piece at The Scholarly Kitchen (TSK) annoyed me a little. The tone implied that the problem with p values a problem in psychology. It isn't. It is a problem in any field that obsesses about p values (including most quantitative work in social sciences, medicine and science). I'm prepared to bet you'd get the same patterns in medical research (and they could well be much worse given some of the pressures to publish positive results in that area). More interesting TSK notes that JEP: General fared better than the other two journals looked at. TSK speculates that this is a result of better editing and reviewing. As it is the only one of the three journals I have reviewed for, I am disposed to agree with this. However, I suspect it may simply be that JEP: General publishes papers with methods that dilute the bias. In particular, I think it publishes more papers that focus on model fit and hence look for p > .05 more often.

The QJEP paper also mentions replicability statistics in passing. Another red herring, I think. Replicability can't be reliably estimated from small samples (see here).

* Actually, if you have to report eta-squared measures you should use generalized eta-squared (see Olejnik & Algina, 2003; Baguley, 2012).

References

Baguley, T. (2012, in press). Serious stats: A guide to advanced statistics for the behavioral sciences. Basingstoke: Palgrave.

Olejnik, S., & Algina, J. (2003). Generalized eta and omega squared statistics: Measures of effect size for some common research designs. Psychological Methods, 8, 434-447.

What's up with social psychology?

2012-08-04T00:08:00.001+01:00

... or to be more precise, what's up with experimental social psychology?

A number of high profile cases of suspected (in some cases admitted) fraud have been highlighted in psychology recently - my own discipline - but they've (nearly all arisen in experimental social psychology. If you are unaware, the best known cases seem to be Diederik Stapel, Dirk Smeesters and now Lawrence Sanna. Another high profile case, Marc Hauser, is in a somewhat related field (but a stretch to call it experimental social psychology). The not so recent case of Karen Ruggiero could also be included.

A separate problem - set aside from the problem of deliberate fraud - are some controversies over specific studies which have apparently hard to replicate effects. The discussion here is around Bargh's priming study and Bem's ESP study. There is ample discussion of this elsewhere, but the main point is that the standard practices in experimental social psychology may encourage publication of spurious effects.

Fraud and other kinds of academic misconduct are rare and far from confined to psychology - see retraction watch (though the scale of Stapel's fraud may have raise psychology's profile on its own). However, the spotlight is focusing heavily on social psychology the moment. My initial view was that experimental social psychology was coming up purely by coincidence, but the recent cases have made me wonder. In the rest of this post I'm going to sketch out some thoughts on what might be going on.

(1) Coincidence. There remains quite good evidence for the whole thing being coincidence. Psychology and social psychology are popular fields with lots of researchers so there will (sadly) be a few frauds. Deliberate fraud is a rare event and (to quote the late, great Robert Abelson) "probability is lumpy". Discrete random events are appear to be evenly or smoothly distributed in the long run (averaging over many events) so rare events are usually clustered in a given sample of small, fixed n. So if you look a 10 or 100 fraud cases there are bound to be clusters among certain disciplines.

(2) Deep discipline-specific flaws. Is there something fundamentally wrong with experimental social psychology research itself? Perhaps. The Bem and Bargh cases point to problems such as lack of replication, pressure to publish, over-emphasis on p values, intolerance of messy data and desire for surprising or counter-intuitive effects. The problem with most of these arguments is that they are not discipline-specific and they are often cited as factors leading to fraud in other disciplines. On the other hand it may be that one or more of these factors are particularly pronounced in experimental social psychology (and I'll come back to this point later).

(3) Enhanced scrutiny. There are three strong reasons to suspect enhanced scrutiny contributes to the recent cases. First, the reports of fraud or other problems are not independent. A case - particularly a big one such as the Diederik Stapel case - necessarily draws further scrutiny to particular journals, groups of researchers and perhaps the whole of a field or discipline. Second, several of the cases were uncovered by the same person: Uri Simonsohn. As Simonsohn works broadly in the area of experimental social psychology, it isn't that surprising that he applied his fraud detection tools to suspicious studies in his own field. Third, findings in experimental social psychology compete for explanatory power more closely with folk psychology explanations than most other fields. To put it another way, just about everyone can assess the plausibility of a study that looks at whether exposing people to old age related material in the lab makes walk faster when they leave the lab or whether eating meat makes you more aggressive. Moreover, they often have strong opinions on these kinds of findings.

At the moment, I think the enhanced scrutiny explanation looks like a strong contender to me. I wouldn't rule out coincidence, but I think we can expect to see a few more dodgy studies unearthed. I think we can also expect to see the label of social psychology expanded to include suspect research in related areas (such as Marc Hauser's work).

Nevertheless, I do think there is one area in which experimental social psychology may be particularly vulnerable to fraud or questionable research practices. High status journals often seek interesting (aka surprising) effects and large effect sizes in the papers they publish. Such findings are more likely to be false (e.g., see here). This is part of a general problem with statistical significance which acts a filter (see Andrew Gelman's blog for lots of discussion on this). A single small experiment can usually only detect relatively 'big' effects - hence it overestimates the size of effects. When you add an implicit requirement for 'big' effects you are biasing your journal or discipline to spurious and fraudulent results. Thus far experimental psychology isn't so different from other fields where small studies are common (e.g., much of medicine, health, neuroscience, biology, and education). The problem may be that effects are inherently smaller in experimental social psychology than other areas of psychology.

I've put the label 'big' in italics because what we're really talking about is the detectability or discriminability of an effect (standardized effect size) - which is its size relative to the noise or error in the data. Experiments with social stimuli are inherently noisy because there are so many variables to control for and because it is often difficult to use big manipulations (as they tend to be pretty obvious to participants). Of course many of the effects may truly be tiny. For example the age priming effect seems plausible to me but I can't believe it would be a large absolute effect in terms of walking speed (easily swamped by other factors or exaggerated by them) - thus my guess is that the original Bargh study over-estimated the effect size (as most early studies tend to).

I think that social psychology and psychology will learn from these cases and the increased scrutiny that seems to be around. I hope we will improve our statistical work, place greater value on replication and reduce the ridiculous pressure to publish ground-breaking, surprising, counter-intuitive work with high frequency. Ground-breaking work will get published, but you can't really tell what research will have real scientific impact until years later (at least two or three years and often much longer, in my view). I hope that psychologists (particularly editors and reviewers) will be more tolerant of messy data (see here) and not quite perfectly watertight conclusions. Many fraudulent studies are detected because of data that are far too clean (real data tend to be messy).

Serious stats book officially published

2012-07-05T12:40:00.001+01:00

My serious stats book is officially published (in the UK at least). The US release date is next month (August 7th). I'm not sure why the release is later (possibly extra shipping time for the books). The earlier European release date is I suppose compensation for the usual pattern for most books and DVDs (where the US is usually first).

More details on the companion blog for the book. The book blog also contains additional resources related to the book and updates on topics covered in the book. Past posts include CIs for differences in correlations, alternatives to the Friedman test, graphing condition means for ANOVA designs, Upcoming plans include (among other things) posts on multilevel ordered logistic regression, p values from multilevel models, order restricted inference ... as soon as I get the time.

p values, the cliff effect and the nature of evidence

2012-06-28T23:53:00.000+01:00

I've been thinking about p values quite a bit recently - prompted by a other bloggers and by some journal work. One interesting phenomenon in this area is the cliff effect: a supposed abrupt shift in researchers' confidence in an effect when moving from p > .05 to p < .05 (assuming that alpha is set at .05, as it usually is).

The p value cliff effect is interesting for a number of reasons (e.g., as a possible cause of dichotomous thinking about effects - see Rosenthal & Gaito, 1963). Personally (e.g., Baguley, 2012a), I think that the tendency to dichotomous thinking is more subtle and complex - and not just a consequence of the cliff effect. Furthermore, there is some doubt about the prevalence of the cliff effect. Poitevineau and Lecoutre (2001) argued that there was a clear cliff effect in only a minority of participants (4 out of 18 in their sample of researchers with PhDs). The appearance of a cliff in the full sample came from averaging over a mix of patterns among the participants - with, for instance, some showing linear increases and some exponential increases in confidence as p decreased.

My own musings on this are about whether there is a 'correct' pattern of confidence in an effect as p decreases. I'm not sure a sensible answer is possible, but I think it is possible to have a stab at answering if you make a few assumptions. First, assume that confidence is driven by the evidential value of a p value. This allows us to draw on the likelihood principle (and hence have some mathematical and philosophical basis for the analysis). I'm not going to explain likelihood or the likelihood principle here - though there is a brief introduction in Chapter 11 of Serious Stats (Baguley, 2012b).

Proceeding on this basis, you hit a major stumbling block. The evidential value in the data is contained in the likelihood function - but the p value doesn't give us the likelihood function (at least, not straight-forwardly). Furthermore, the law of likelihood implies that you can't get the evidence for an effect from either p or the likelihood. Evidence for a specific hypothesis (e.g., a null hypothesis such as mu = 0) can only be assessed relative to a specific alternative hypothesis (e.g., mu = 10). So even if you could get at the likelihood function you'd need to know what alternative hypothesis each researcher had in mind when they evaluated p. To make the problem tractable you could assume that the assume the alternative hypothesis at the most likely value (the maximum likelihood estimate). For simple situations this boils down to the observed parameter estimate of interest (e.g., a mean or difference in means). This is helpful because you can go from a p value derived from a z statistic to an approximate maximum likelihood ratio fairly easily (with additional assumptions about the asymptotic distribution of the test statistic).* The maximum likelihood ratio is the largest possible ratio of the evidential support in favour of the alternative hypothesis relative to the null hypothesis (of zero effect) from the data at hand (ignoring support from other sources). This is the most favourable assessment of the evidence for the alternative hypothesis and against the null on the basis of the data alone (and true evidential support is almost certain to be lower).

Plotting the maximum likelihood ratio as a function of p gives the following curve:

It flattens out when p > .20, so I've just shown the 'curvy' bit. Of course there is no particular reason why you'd use likelihood ratio ratio than say probability of the null hypothesis being false:

This shows a more gradual curve. Yet another possibility is the logarithm of the maximum likelihood ratio (commonly used to index evidential support in statistics):

Each of these is a plausible quantity to drive intuitions of evidential support (though you could easily arrive at different curves by making different - and perhaps more plausible - assumptions).

The upshot is that neither a fairly abrupt cliff effect, nor a smooth (near linear) function nor a steep exponential-style curve would be unreasonable. It is also clear that the task itself - assessing confidence in an effect - is far from trivial. Almost any continuous monotonic function could be considered as rational given the right assumptions and there is no particular reason to expect researchers to be able to do the right computations in their heads ...

On the other hand, a step function cliff effect does seem unreasonable - and it is definitely still interesting to understand the psychology of expert and non-expert statistical reasoning. However, we should be wary of assuming that people are being irrational or are bad at statistical reasoning on the basis of this kind of task.

Post script. This is all predicated on a few idle speculations, several highly debatable assumptions and a quick play with some R code. It is quite possible I've overlooked something major ...

* see Edwards et al. (1963) or Serious Stats Chapter section 11.4.2.

References

Baguley, T. (2012a). Can we be confident in our statistics? The Psychologist, 25, 128-9.

Baguley, T. (2012b). Serious stats: A guide to advanced statistics for the behavioral sciences. Basingstoke: Palgrave.

Edwards, W., Lindman, H., & Savage, L. J. (1963). Bayesian statistical inference for psychological research. Psychological Review, 70, 193-242.

Poitevineau, J., & Lecoutre, B. (2001). Interpretation of significance levels by psychological
researchers: The .05 cliff effect may be overstated. Psychonomic Bulletin & Review, 8,
846-850.

Rosenthal, R. & Gaito, J. (1963). The interpretation of levels of significance by psychological researchers. Journal of Psychology, 55, 33-38.

Rosenthal, R. & Gaito, J. (1963). The interpretation of levels of significance bypsychological researchers. Journal of Psychology, 55, 3338.

The stimuli-as-fixed-effect fallacy

2012-06-21T16:16:00.001+01:00

Neuroskeptic has just blogged on a new paper by Judd, Westfall and Kenny on Treating stimuli as a random factor in social psychology: A new and comprehensive solution to a pervasive but largely ignored problem. I can't access the original paper (which is supposed to be available via my University but hasn't appeared yet ...) but I know a little bit about the topic and thought I'd write a few words.

What stimulated me to write was a) a few of the comments on Neuroskeptic's blog, and b) that I've just written a book that covers the topic in some detail. (Yes - this book!).

The basic problem is that standard statistical analyses in psychology treat participants (subjects) as a random factor, but stimuli as a fixed factor. Thus our statistics assume that the goal of inference is to say something about some population that those participants are representative of (rather than just the particular people in our study). By treating stimuli as fixed it is assumed that we've exhaustively sampled the population of interest in our study. This limits statistical generalization to those particular stimuli. This is an unattractive property for psycholinguists (because they tend to be interested in, say, all concrete nouns rather than the 30 nouns used in the study). The same issue may apply to lots of other types of stimuli (faces, people, voices, pictures, logic problems and so forth).

The comments fell into several camps, but one response was that this was another case of researchers getting basic stats wrong. I consider this to be unfair because we're not talking basic stats here. The problem is quite subtle and the solutions are, in statistical terms, far from basic. Furthermore, it is not always an error. There are situations in which you don't need to worry about the problem and situations in which it is debatable what the correct approach is.

Another response was the psycholinguists have known about this problem for years (true!) and have analyzed their data correctly (false!). The problem came to prominence in a paper by Herb Clark (The language-as-fixed-effect fallacy), but was originally raised by Coleman (1964). Clark noted that running separate ANOVAs treating subjects as unit of analysis and items as unit of analysis did not solve the problem (by-subject and by-item analyses). Either analysis is statistically non-significant the effect fails to generalize, but if both are statistically significant the correct analysis (that combines variability across subjects and items) might still be statistically non-significant. His solution was to estimate the correct ANOVA test statistic (quasi F or F') with a simple-to-calculate minimum value (min F'). This is known to be conservative (i.e., produces p values that are slightly too large) but not unreasonably so in practice (see Raaijmakers et al., 1999). Raaijmakers et al. (1999) show that until recently most psycholinguistic researchers still got it wrong (e.g., by reporting separate by-item and by-subject analyses).

What is the correct approach? Well, it depends. First, do you need to generalize beyond your stimuli set? This has to do with your research goals. In some applied research you might just need to understand how people respond to a particular set of stimuli. A single stimulus or stimulus set can offer a counterexample to a strong claim (e.g., that X is always the case). Alternatively, it might be reasonable to assume that the stimuli are - for the purposes of the study - very similar to others in the population (i.e., that population variability is negligible). This might be the case for certain mass-produced products (e.g., brands of chocolate bar) or precision-engineered equipment. However, a lot of the time you do want to generalize beyond your sample of stimuli ...

That leaves you with the option of altering the design of the study or doing incorporating the extra variability owing to stimuli into the analysis. The design option was considered by Clark (1973) and by Raaijmakers et al. (1999). Clark pointed out that if each person had a different (ideally random) sample of items from the stimulus population then the F ratio of a conventional ANOVA would be correct. The principle here is quite simple: all relevant sources of variability need to be represented in the analysis. By varying the stimuli between participants the variability is present and ends up being incorporated into the between-subjects error term.* This is quite a neat method and can be easy to set up in some studies (e.g., if you have a very large pool of words to sample from by computer). Raaijmakers et al. (1999) also notes that you get the correct F ratios from certain other designs. This, in my view, is only partly true. Any design that restricts the population sampled from (of participants or stimuli) restricts its variability and therefore restricts its generalizability to the pool of participants or stimuli being sampled from.

Recent development in statistics and software (or at least recent awareness of them in psychology) have brought the discussion of the language-as-fixed-effect fallacy or more properly stimuli-as-fixed-effect fallacy back to prominence. In principle it is possible to use a multilevel (or linear mixed) model to deal with the problem of multiple random effects (and this has all sorts of other advantages). However, the usual default model is a nested model that implicitly assumes that stimuli presented to each person are different.

A nice point here is that a nested multilevel repeated measures model fitted with RML (restricted maximum likelihood) and a certain covariance structure (compound symmetry) is pretty much equivalent to repeated measures ANOVA and can be used to derive standard F tests etc. Thus Clark's assertion about using a design with stimuli nested within participants producing the correct F ratios is confirmed.

Baayen et al. (2008) offered a critique of the standard approach and explained how to fit a multilevel model with crossed random factors (i.e., where stimuli are the same for all participants ... or equivalently participants are the same for all stimuli). These models can be fit in software such as MLwiN or R (but not SPSS**) that allows for cross-classified multilevel. The lme4 package in R is particularly useful because it fits these models fairly effortlessly.

This looks to be the solution described by Judd, Westfall and Kenny - as far as I can tell by their abstract and the solution I cover in my book (Baguley, 2012).

* Note that a by-item analysis or by-subject analysis violates this principle because the each analysis uses the average response (averaged over the levels of the other random factor) and the variability around this average is unavailable to the analysis.

** UPDATE: Jake Westfall kindly sent me a copy of the paper. I have not read it properly yet but looks extremely good. He points out that recent versions of SPSS can run cross-classified models (I'm still on an older version). Their paper includes SPSS, R and SAS code. I would still recommend R over SPSS. One highlight is that show how to compute the Kenwood-Roger approximation in R. Complex multilevel models make it difficult to assess the correct df for effects and the Kenwood-Roger approximation is one of the better solutions. In my book I used parametric boostrapping or HPD intervals to get round this problem, but this is potentially a very useful addition.

References

Baayen, R. H., Davidson, D. J., & Bates, D. M. (2008). Mixed-effects modeling with crossed random effects for subjects and items. Journal of Memory & Language, 59, 390-412.

Baguley, T. (2012). Serious stats: A guide to advanced statistics for the behavioral sciences. Basingstoke: Palgrave.

Clark, H. H. (1973). The language-as-fixed-effect fallacy: A critique of language statistics in psychological research. Journal of Verbal Learning and Verbal Behavior, 12, 335-359.

Coleman, E. B. (1964). Generalizing to a language population. Psychological Reports, 14, 219-226.

Raaijmakers, J. G. W., Schrijnemakers, J. M. C., & Gremmen, F. (1999). How to deal with "The language-as-fixed-effect fallacy": Common misconceptions and alternative solutions. Journal of Memory & Language, 41, 416-426.

The aesthetics of error bars

2012-05-27T21:28:00.000+01:00

This blog and my other main blog (the companion blog for my book) are now syndicated via R-bloggers (posts tagged R only) and statsblogs.com. The latter is a relatively new blog aggregator but looks to have some interesting content. R-bloggers it quite well established and I was already an occasional reader.

Looking at some recent content I noticed an interesting piece by Ben Bolker (author, among other things, of the excellent bbmle package in R) on dynamite plots. Until a few years ago (possibly in the early research for my book) I had not heard the term 'dynamite plot' or the negative press the attract in some research fields. In my own discipline (psychology) and in experimental psychology in particular bar plots with error bars (looking like sticks of dynamite stacked in a row) are rather popular. In fact, I was taught to use them in preference to dot plots when plotting interactions in ANOVA (their main application in experimental psychology). The main arguments against dot plots are that it is easy to manipulate them to make effects look large large by adjusting the scale (and sometimes software does this automatically). The advantage of switching to a bar plot is that these are supposed to be zero-referenced and thus effects are likely to more appropriately scaled.

Here is an example of a dynamite plot adapted from chapter 3 of my book:

(Colleagues of mine will note that the quantity displayed by the error bars is not labeled. This should always be clear from plot or figure caption - here they are 95% CIs).

Some of the material on dynamite plots on the web is somewhat one-sided (e.g., see here, here, here or the comments here). Ben Bolker bravely presents a more balanced picture. He also gets to the heart of the issue by noting that most criticisms of dynamite plots suggest box plots or plots of raw data as alternatives. This doesn't seem appropriate if your goal is inference rather than description. As Bolker notes, if you've decided to something like ANOVA you are already implicitly assuming approximate normality of the errors and so forth. Thus if the main purpose of the plot is inferential or to display key patterns among the data, box plots or raw data plots are not so useful. (Don't get me wrong I think think that plotting raw data is a good idea - but exploratory work and model checking are different from inference). So for a plot of means with error bars, the choice of dot plot or bar plot is one of aesthetics. These days my preference is for dot plots (which are more versatile and have a better information to ink ratio), but I think a well constructed dynamite plot can be appropriate in some situations. I would usually save these for a situation in which the pattern was quite simple (e.g., a 2 by 2 interaction), there was a meaningful zero or other reference point and when my audience are familiar with this style of plot and may prefer them.

A further aesthetic point here is how to plot the error bars themselves. I am persuaded by Andrew Gelman's argument that the crossbars on conventional error bar plots are ugly and counterproductive. They draw your attention to the extremes of the error bar - when values closer to the statistic being estimated are more plausible. Here is the earlier dyamite plot redrawn as a conventional error bar plot and in cleaner Gelman-approved style:

I find the version on the right to be much prettier. Furthermore it makes it easier to adapt them into two-tiered error bar plots. I like to use two-tier plots to convey 95% CIs for individual means (outer tier) and inferential (difference-adjusted) 95% CIs (inner tier). The inner tier approximates to a 95% CI for the difference - so that the means can be considered different by conventional criteria if the inner tier error bars don't overlap:

In my paper on within-subject CIs I used the style on the left. However, with hindsight I wish I'd included the style on the right. Varying the width of the bars avoids the ugly crossbars but may make detecting a 'statistically significant' difference trickier. I think that aesthetics win here because graphical methods aim to support informal inference - they are not supposed to be there for fine-grain, formal inference (which can be supported by formal hypothesis tests of various kinds - not just null hypothesis significance tests).

UPDATE: The functions for these plots are on the book blog. More generally my functions for the book, CIs for ANOVA and a few other things are all available here. I plan to update these functions regularly to add functionality and deal with any undocumented features.

Serious stats - free statistics resources

2012-03-23T20:52:00.000Z

The companion web site for Serious Stats is now live:

http://www.palgrave.com/psychology/baguley/

The web site includes:

- a free sample chapter (Chapter 15: Contrasts)
- data sets
- R scripts
- 5 online supplements (for meta-analysis, multiple imputation, replication probabilities, pseudo-R squared and loglinear models)

Also don't forget the Serious stats blog to accompany the book.

Graphing between-subject confidence intervals for ANOVA

2012-03-19T11:58:00.002Z

This is a quick follow up to my earlier post that discussed how to graph CIs for within-subjects (repeated measures) ANOVA designs. My forthcoming book Serious stats describes how to do this for between-subjects designs (a much simpler problem). The blog that accompanies the book now has a post summarizing the main options and explaining how to plot difference-adjusted CIs (95% CIs constructed so that non-overlapping intervals correspond to a statistically significant difference between means at p < .05). In addition, the post includes R functions to calculate and plot difference-adjusted CIs (though the calculations are not difficult to reproduce by hand).

UPDATE: I've now added functions for two-tiered CIs for between-subjects designs on the book blog. More generally my functions for the book, CIs for ANOVA and a few other things are all available here. I plan to update these functions regularly to add functionality and deal with any undocumented features.

p curves revisited

2012-03-15T23:35:00.000Z

I finally found some time to take a closer look at p curves. I haven't had a chance to follow-up my simulations (and probably won't for a few weeks if not months), but I have had time to think through the ideas the p curve approach raises based on some of the comments I've received and a brief exchange with Uri Simonsohn (who has answered a few of my questions).

First, I got a couple of things at least partly wrong.

i) how p curves work

ii) the potential for correlated p values

How p curves work

I made the (I think) reasonable assumption that p curve analysis involved focusing on a bump just under the p = .05 threshold. Other work (Wicherts et al., 2011) has shown that there is indeed some distortion around this value. My crude simulation suggested that p curves could maybe be used to detect this kind of bump - but that the method was noisy and required large N.

All good so far except my assumption was completely wrong. This isn't what Simonsohn and colleagues are proposing at all. They are focusing on the whole of the distribution between p = 0 and p = .05. This is a very different kind of analysis because it uses all the available p value information about 'p hacking' (if you accept the highly plausible premise that p hacking is concentrated on statistically significant p values).

Null effects will therefore produce a flat p curve (because the distribution of p under the null is uniform). Simonsohn argues that non-null effects should produce downward sloping p curves. He and his colleagues have simulated p curves under various ranges of effect size to confirm this - and there is also an analytic proof for the normal case (Hung et al., 1997).* I also (inadvertently) confirmed this in my original simulations - which show the downward sloping trend (but note that I include p values up to p = .10 in my plots).

However, mixing in p hacked studies to a flat curve will produce an upward sloping curve - the feature that Simonshohn and his colleagues are focusing on. I haven't simulated this directly - but it seems sensible because p hacking is (in essence) a flavour of optional stopping (adding data or iterating analyses until you squeeze a statistically significant effect out). Certainly, an upward sloping curve would be a signal of something wierd going on.

This approach uses more information than my mistaken 'p bump' approach and so should be much more stable.

* It is far from unreasonable to treat the distribution of effects as approximately normal - as is common in meta-analysis (and see also Gillett, 1994), but I don't think the pattern depends strongly on this assumption.

Correlated p values

It is well known that p values are inherently extremely noisy 'statistics' - they jump around all over the place for identical replications. Geoff Cumming and colleagues have published some good work on this (e.g., Cumming & Fidler, 2009). Thus the same effect in different studies or different effects of similar sizes will in general not tend to have correlated p values. However, the noise that causes this jumping around will be crystalized if you use the same data to re-calculate the p value. This could cause correlated p values where data is re-used or where variables are very highly correlated. For example, this could happen if you add a covariate that is a modest predictor of Y and uncorrelated with and report p values with and without the covariate. It could also happen if you report essentially the same analysis twice with a very similar variable (e.g., X correlated with children's age or X correlated with years of schooling).

There are two main solutions here: a) just filter out p values that re-use data or use highly-correlated data, or b) model the correlations in some way by accounting for within-study clustering - as you might in a multilevel model and some forms of meta-analysis (itself a form of multilevel model).

In summary, I think the p curve approach looks very interesting, and I'd certainly like to see more work on it (and hope to see the full version published some time soon).

References

Cumming, G., & Fidler, F. (2009). Confidence Intervals. Zeitschrift für Psychologie / Journal of Psychology, 217(1), 15-26.

Gillett, R. (1994). Post hoc power analysis. Journal of Applied Psychology, 79(5), 783-785. doi:10.1037//0021-9010.79.5.783

Hung, H. M., O’Neill, R. T., Bauer, P., & Köhne, K. (1997). The behavior of the P-value when the alternative hypothesis is true. Biometrics, 53(1), 11-22.

Wicherts, J. M., Bakker, M., & Molenaar, D. (2011). Willingness to share research data is related to the strength of the evidence and the quality of reporting of statistical results. PloS one, 6(11), e26828. doi:10.1371/journal.pone.0026828

R code for p curves

2012-03-14T20:23:00.000Z

I have finally got around to posting the R code for my p curve simulation. Those familiar with R will realize how crude it is (I've been caught up with other urgent stuff and had no time to explore further).

You are welcome to play with (and improve!) the code. Changing delta will alter the (at present) fixed effect size. It would be more realistic to vary this (and the sample sizes). A good starting point for the effect size distribution (in the population) might be a normal distribution with say a mean of zero and a variance of 1 (see Gillett, 1994).

delta <- 0.5
m1 <- 10
sd <- 2
m2 <- m1 + sd*delta
n1 <- n2 <-25
 
n.sims <- 500
p.data <- replicate(n.sims, t.test(rnorm(n1, m1,sd), rnorm(n2, m2,sd))$p.val, simplify=T)
 
par(mfrow=c(5,3))
for (i in 1:15) {
 p.data <- replicate(n.sims, t.test(rnorm(n1, m1,sd), rnorm(n2, m2,sd))$p.val, simplify=T)
 hist(p.data, xlim=c(0,0.1), breaks = 99, col = 'gray')
 }

R code html script courtesy of Pretty R at inside-R.org

References

Gillett, R. (1994). Post Hoc Power Analysis. Journal of Applied Psychology, 79, 783-785.

Simulating p curves and detecting dodgy stats

2012-02-16T21:08:00.000Z

Psych your mind has an interesting blog post on using p curves to detect dodgy stats in a a volume of published work (e.g., for a researcher or journal). The idea apparently comes from Uri Simonsohn (one of the authors of a recent paper on dodgy stats). The author (Michael W. Kraus) bravely plotted and published his own p curve - which looks reasonably 'healthy'. However, he makes an interesting point - which is that we don't know how useful these curves are in practice - which depends among other things on the variability inherent in the profile of p values.

I quickly threw together a simulation to address this in R. It is pretty limited (as I don't have much time right now), but potentially interesting. It simulates independent t test p values where the samples are drawn from independent, normal distributions with equal variances but different means (and n = 25 per group). The population standardized effect size is fixed at d = 0.5 (as psychology research generally reports median effect sizes around this value). Fixing the parameters is unrealistic, but is perhaps OK for a quick simulation.

I ran this several times and plotted p curves (really just histograms with bins collecting p values at relevant intervals). First I plotted for an early career researcher with just a few publications reporting 50 p values. I then repeated for more experienced researchers with n = 100 or n = 500 published p values.

Here are the 15 random plots for 50 p values:

At least one of the plots has a suspicious spike between p = .04 and .05 (exactly where dodgy practices would tend to push the p values).

What about 100 p values?

Here the plots are still variable (but closer to the theoretical ideal plotted on Kraus' blog).

You can see this pattern even more clearly with 500 p values:

Some quick conclusions ... The method is too unreliable for use with early career researchers. You need a few hundred p values to be pretty confidence of a nice flat pattern between p = .01 and p = .06. Varying the effect size and other parameters might well inject further noise (as would adding in null effects which have a uniform distribution of p values and are thus probably rather noisy).

I'm also skeptical that this is useful for detecting fraud (as presumably deliberate fraud will tend to go for 'impressive' p values such as p < .0001). Also (going forward) fraudsters will be able to generate results to circumvent tools such as p curves (if they are known to be in use).

On nonparametric statistics ...

2012-02-14T23:54:00.003Z

I'm not a big fan of the term "nonparametric statistics", or at least how it is used in psychology and related fields (e.g., education and health research). This is one reason why I don't make a big deal of the parametric/non-parametric distinction in my Serious stats book and probably partly why a recent article in APS Observer annoyed me so much.

Why did it annoy me so much? It says, in essence, what many standard psychology text books say (and also makes several good points - which I shall largely ignore for rhetorical purposes). I like in particular the points about statistical tests used to check statistical assumptions lacking power and making statistical assumptions themselves (which are typically unchecked and they are not robust to). Indeed, that's another thing that annoys me and I have blogged/bored people about in the past (e.g., see here). I also think the article might have been chopped about a bit by editors (as it contains at least one important reference not cited and some of my gripes are also contained in references cited by the article I'm annoyed with). So, on balance, it is (by psychology mainstream standards) not at all a bad piece. However, by buying into the standard nonparametric stats presentation it perpetuates a few myths or errors and may inadvertently gives some pretty poor advice mixed in with the good.

Here is a quick list of my criticisms:

(1) The definition of nonparametric statistics is deeply confused. The author starts by writing: "Nonparametric statistical analyses are used to investigate research questions in which the dependent variable is ranked or categorical rather than quantified in a true numeric sense". This seems to suggest that non-parametric procedures are defined by having a discrete or bounded DV. Later it adds that "Traditional parametric statistics require a number of assumptions about the characteristics (i.e., parameters) of the data." This is an appeal to the idea that parametric statistics assume a particular probability distribution (the parameters of which are estimated by the data). This seems like a better definition to me, although like many psychologists, the author appears to assume that the probabilty distribution assumed is always a normal distribution. Mixing the two aspects of the definition is confusing. It is easy to find statistical procedures that are parametric in the second sense, but involve ranked or categorical DVs. I would argue that a chi-square test of independence or a sign test is parametric by the second definition. Not only is the definition problematic, but it could lead to poor analytic decisions. For instance, dichotomous outcomes are often best analyzed using parametric techniques such as logistic regression (a generalized linear model with a logistic link function and a binomial random component) rather than the methods surveyed in the article.

(2) The article also reads as if nonparametric tests are the same thing as rank randomization tests. Rank randomization tests are examples of nonparametric tests in the sense they are distribution free (making no assumption about a particular probability distribution for the data). However, there are many other nonparametric methods that don't involve ranks. Rank randomization tests are useful but limited in scope. In addition, if the raw data are in the form of ranks then a rank transformation is pretty pointless (you might as well jut run a regular test).

The main limitation of the rank approaches is that the rank transformation is irreversible; it destroys the link between the analysis and the raw scores. This is a serious problem if you care about the raw scores - perhaps because you want to test whether effects are non-additive or because you want to get an interval estimate or a effect sizes on the original scale. There are ways to help you do this with rank transformation procedures, but they are generally pretty fiddly. There are good reasons why people don't tend to use rank transformations in multiple regression, use them to test interaction effects or use them to construct confidence intervals.

Third, rank randomization (and other nonparametric) procedures are bound by pesky assumptions! They tend to make weaker assumptions than parametric procedures, but all statistical procedures make assumptions. The assertion that the "main assumptions of nonparametric tests are that the dependent variable should be continuous and have independent random sampling, which means that nonparametric statistics do not require assumptions of homogeneity of variance and normality" is misleading. The precise assumptions vary from test to test and with the hypothesis being tested. As a rule, rank randomization and related rank transformation tests assume that samples with similar shapes of distribution are being compared. They can therefore be undermined by heterogeneity of variance or varying degrees of skew. If the distributions have very dissimilar shapes then the tests can sometimes behave very strangely (e.g., it is possible to get outcomes where A > B, B > C and yet C > A). Continuity is also not necessarily a requirement of the DV or of the underlying construct being measured (though it may be for some hypotheses).

(4) Many generalizations are made that simply don't hold up: "When the data violates the assumptions of a parametric test, nonparametric tests are again the more powerful analytic technique (Siegel, 1957)." This doesn't follow. Often this is true, but not always. There has been a lot of research on this topic since 1957 and parametric tests are not always inferior. This is particularly true if you don't restrict parametric techniques to t tests, linear regression and ANOVA. Some nonparametric techniques are known to have very poor power. I also got irritated by the critique of log and similar transformations that states: "while these transformations can make variables more normally distributed, they can also diminish or alter experimental effects, which can reduce power." To the extent this is true, it is also true of the rank transformation (more so in some cases). Furthermore, the real problem is with arbitrary transformations. Log transformations - where appropriate - tend to aid interpretation of effects (e.g., by quantifying them as proportionate rather than additive effects). I also dislike the implication that "experimental effects" exist in a pristine form prior to transformation. This is simply not the case - how to quantify and scale measurement of an effect is a tricky business (e.g., memory researchers can use percentage correct, hits minus false alarms, A prime d prime etc.) and many effects come "pre-transformed" (e.g., measurement of loudness in decibels). Transformations (including the rank transformation) are useful tools that can increase power and aid interpretation if used carefully and appropriately.

(5) The use of parametric statistics to analyze Likert-style rating scales may be one of the "seven deadly sins of statistical analysis", but it is rarely a big problem in practice. Least squares methods are most messed up by heavy tailed distributions, severe skew or outliers. If anything, Likert-style rating scales tend not to have these problems (or to manifest them relatively mildly). Furthermore, where there are problems with Likert-style measures, rank randomization or transformation tests s are probably not the solution. A number of parametric procedures for ordinal outcomes exist - notably ordered logistic regression (though least squares methods such as t tests, ANOVA or regression should work well when their assumptions are not badly violated).

(6) The variety of nonparametric tests referred to is slightly artificial. As a general rule there are advantages to sticking to standard 'parametric' tests such as the Welch-Satterthwaite t test or one-way ANOVA rather than using named rank-transformation tests such as the Mann-Whitney U test. In some cases there may be advantages with specialized rank randomization tests where sample sizes are small (e.g., because software such as R implements exact versions). However, there are a few cases where the rank randomization tests are not robust (e.g., Mann-Whitney U test is not robust to heterogeneity of variance) or lack power (e.g., the Friedman test, Page's L test and most multiple comparison procedures available for ranks). Rank transforming the data and then running a t test with Welch-Satterthwaite correction is superior to running the Mann-Whitney directly (Zimmerman & Zumbo, 1993a). For more on the low power of the Friedman test and better alternatives see here.
Rather than think in terms of nonparametric statistics, it is better to focus on checking assumptions (using graphical methods and simple descriptives) and checking our models against more robust procedures. If more robust methods show different results - the next step is to find out why (and definitely not report just the outcome you prefer). This should lead you to a superior model (using robust methods or perhaps a more appropriate parametric model). The consideration of robust methods is particularly important. This includes some rank transformation tests, but also includes robust regression, bootstrapping and other tools (e.g., see Wilcox & Keselman, 2003; 2004; Baguley, 2012).

References

Baguley, T. (2012, in press). Serious stats: A guide to advanced statistics for the behavioral sciences. Basingstoke: Palgrave.

Wilcox, R. R., & Keselman, H. J. (2003). Modern robust data analysis methods: Measures of central tendency. Psychological Methods, 8, 254-274.

Wilcox, R. R.,& Keselman, H. J. (2004). Robust regression methods: Achieving small standard errors when there is heteroscedasticity. Understanding Statistics, 3, 349- 364.

Zimmerman, D. W., & Zumbo, B. D. (1993b). Rank transformations and the power of the Student t test and Welch t' test for non-normal populations with unequal variances. Canadian Journal of Experimental Psychology, 47, 523-539.

Comparing correlations update

2012-02-05T17:42:00.002Z

I have just published R code for calculating CIs for differences between correlations on the Serious stats book blog. This covers independent correlations (taken from chapter 6 of the book) and dependent correlations (new R code written as a supplement to chapter 6).

UPDATE on the update ...

I have also added an Excel spreadsheet that should match the R output (though the latter is probably more accurate and reliable).

Serious Stats book and blog update

2012-02-02T16:50:00.001Z

This is a quick update to announce my new blog Serious Stats. This is a companion to my forthcoming book of the same name:

Baguley, T. (2012, in press). Serious stats: A guide to advanced statistics for the behavioral sciences. Basingstoke: Palgrave.

I think is better to separate book specific content out from my regular posts (though in some cases this will be a bit fuzzy). I will also try and post short updates here when something relevant gets published on the blog for the book.

More on "A problem of significance"

2012-02-01T20:38:00.009Z

A longer version of my earlier post A problem of significance just appeared in The Psychologist.

Baguley, T. (2012). Can we be confident in our statistics? The Psychologist. 25, 128-9.

Shortly after publication I received an email asking about statistical analysis of differences in correlations. This is more tricky than you might think. I'm working on some R code to implement one of the better approaches and plan to blog on this shortly ...

(See update here.)

Calculating and graphing within-subject confidence intervals for ANOVA

2011-10-04T15:43:00.006+01:00

Psychologists are gradually coming round to the view that it is a good idea to present interval estimates alongside point estimates of statistics. The most common statistic reported in psychology research is almost certainly the mean (strictly the arithmetic mean). Presenting an interval estimate for the mean of a single sample is usually quite simple. This is usually done as 95% confidence interval about the mean – and most researchers in psychology are able to calculate this by hand or get their statistical software to calculate and graph it for them.

Extending this to more than one mean introduces an additional layer of complexity. This is because the difference between two means is a different quantity, and its CI (although related to those of the individual means) is different in width from the CIs of the individual means. This creates a problem when plotting the CI because a researcher might be interested in the CI for an individual mean, the CI for their difference (or both).

The complexity increases further if the aim is to plot a set of means (e.g., from an ANOVA design). In this case, plotting all the possible differences (as is commonly done) obscures patterns in the individual means (e.g., linear or quadratic trends). Last, but not least, if the means are not from independent samples, there are further difficulties. This happens in within-subjects or repeated measures designs.

In these designs the variation around each mean is correlated with the variation around the other means. This correlation arises from individual differences. Statistical procedures such as ANOVA can capitalize on these individual differences to produce more sensitive statistical inferences (i.e., to increase statistical power or obtain narrower CIs). This is done by estimating the variation due to individual differences, and removing it from the error variance (the estimate of statistical noise in the data set).

This is a problem for graphical presentation of means because the precision of individual means is influenced by individual differences, whereas the precision of differences between means is not (because the estimate of individual differences is common to repeated samples from the same people and thus can be removed). Further complications arise when the sphericity assumption of repeated measures ANOVA is violated.

Several solutions to these problems have been proposed in the literature. The best known of these in psychology is that of Loftus and Masson (1994). Another well-known solution is that of Goldstein and Healy (1995), extended to correlated samples by Afshartous and Preston (2010).

Despite a large literature on the problems of graphing a set of correlated means, many people avoid the problems altogether by not reporting (or graphing) CIs or report CIs that are misleading in some way. Researchers are often unaware of the problems or find the solutions hard to understand and implement.

I recently reviewed the main approaches in the literature, describe how to obtain suitable intervals for individual means and differences between means and provide R code to calculate and plot the intervals.

The main highlights are that:

i) for inferences about individual means the standard approach works fairly well for between-subject (independent measures) designs, but there is a case to use CIs from a multilevel model for within-subject (repeated measures) designs

ii) an approach proposed by Cousineau (2005) with a correction by Morey (2008) offers advantages over the Loftus and Masson (1994) approach for within-subject ANOVA designs. It simplifies the calculations and does not assume sphericity. The Loftus-Masson approach will however usually be superior when n is small.

iii) if you are interested in differences between means then you should probably plot a version of the Cousineau-Morey (or Loftus-Masson) interval that is adjusted so that overlap of the CIs around two individual means corresponds to overlap of the CI for their difference. This can be done by incorporating a multiplier to the width of the individual CIs. This multiplier is equal to (2^0.5)/2.

iv) if you are interested in both precision of individual means and their differences you can use a two-tiered error bar to display both quantities (Cleveland, 1985).

v) the intervals (and graphical presentation of means) are useful for informal inference about a set of means. For formal inference it is better to set up precise hypotheses and test these via an a priori of contrast. This could be a traditional null hypothesis significance test, but other approaches are available. These include confidence intervals, Bayes factors, likelihood ratios and so forth (Baguley, in press; Dienes, 2008).

The paper is available here, the R code here and the data sets here.

Update: R functions now available for the simpler between-subjects (independent measures) ANOVA
case (at the Serious stats blog).

References

Afshartous D., & Preston R. A. (2010). Confidence intervals for dependent data: equating nonoverlap with statistical significance. Computational Statistics and Data

Analysis. 54, 2296-2305.

Baguley, T. (2011, in press). Calculating and graphing within-subject confidence intervals for ANOVA. Behavior Research Methods. DOI: 10.3758/s13428-011-0123-7

Baguley, T. (2012, in press). Serious Stats: A guide to advanced statistics for the behavioral sciences. Basingstoke: Palgrave.

Cleveland. W. S. (1985). Elements Of Graphing Data. New York, NY: Chapman & Hall.

Cousineau, D. (2005). Confidence intervals in within-subject designs: A simpler solution to Loftus and Masson’s method. Tutorials in Quantitative Methods for Psychology, 1, 42-45.

Dienes, Z. (2008). Understanding Psychology as a Science: An Introduction to Scientific and Statistical Inference. Basingstoke: Palgrave Macmillan.

Goldstein, H., & Healy, M. J. R. (1995). The graphical presentation of a collection of means. Journal of the Royal Statistical Society. Series A (Statistics in Society), 158, 175-177.

Loftus, G. R., & Masson, M. E. J. (1994). Using confidence intervals in within-subject designs. Psychonomic Bulletin & Review, 1, 476-490.

Morey, R. D. (2008). Confidence intervals from normalized data: A correction to Cousineau (2005). Tutorials in Quantitative Methods for Psychology, 4, 61-64.

A problem of significance

2011-09-15T23:27:00.006+01:00

Several people have drawn my attention to a recent article on a common error in published statistical analyses in neuroscience. Sander Nieuwenhuis, Birte Forstmann and Eric-Jan Wagenmakers published (in Nature Neuroscience) a critique of statistical analyses in the neuroscience literature. This paper has been written about by Ben Goldacre and Andrew Gelman (who published an article on the general problem some time ago) - so I won't go into too much detail.

The point of interest for me is that the error concerns something that most psychologists should know all about (and hence should be expected not to make the error). It concerns the case of two differences, one statistically significant and one non-significant. For example, group 1 may show a significant difference between experimental condition and placebo (for a drug intervention), while group 2 do not. A naive interpretation is that the drug works for group 1 but not group 2. This is not necessarily true. The proper test of a difference in effects of the drug between groups is an interaction test. Psychologists tend to avoid this error because we have heavily trained in ANOVA as undergraduates (certainly in the UK and probably also in the US and most of Europe). Even if we fail to learn this, reviewers and editors (in psychology) tend to spot the error.*

Are psychologists then entitled to feel a little bit smug? Perhaps, but only a little. First, I think the reason we are relatively good performers on this point is because we tend to view many statistical analyses through an "ANOVA" lens. Factorial ANOVA (in which factors are orthogonal) includes the interaction term by default. The 2 by 2 factorial ANOVA is the workhorse of experimental psychology. Our familiarity with this type of design and analysis makes this easy to spot. Second, our ANOVA lens leads to other errors - notably dichotomizing continuous variables (e.g., via median split) in order to squeeze them into an ANOVA design. This always decreases statistical power, and can - albeit infrequently - produce spuriously significant effects (see MacCallum et al., 2002). These errors are sometimes less serious than the difference of differences/interaction error (but are not harmless).

The real test then, is whether psychologists make the same (conceptual) error in a different context. The obvious context is that of association rather than difference. If males show a significant correlation between testosterone and aggression (e.g., r = .5, N = 25) and females don't (e.g., r = .3, N = 25), the correlation between testosterone and aggression is not significantly bigger for males than females. To confirm this you'd need to construct a test or (better still) confidence interval for the difference in correlations. This is hardly ever done - and, in my experience, psychologists frequently make this kind of claim without backing it up.** Methods for testing differences in correlations are a bit fiddly (e.g., depending on overlap or lack of overlap in the measurements), and rarely taught at undergraduate or even postgraduate level. The methods that are taught are also often a bit dodgy (see Zou, 2007, for some better alternatives).

Also note that (in both cases) the error can work the other way. Two correlations could be non-significantly different from zero but different from each other (e.g., r = .5 and r = -.5 with N = 10).

Postscript

There is, I think, a lesson or two here. A minor lesson is that interactions are bit more complicated than psychologists (particularly those very familiar with ANOVA) often think. I could write more on this (and do a bit in my forthcoming book). A major lesson is that this concept (the difference between significant and non-significant is not necessarily also statistically significant - see Gelman & Stern, 2006) is probably quite tricky. It may be worth exploring why ... I suspect it is because of several factors.

See updates here and here.

References

Gelman, A., & Stern, H. (2006). The difference between “significant” and “not significant” is not itself statistically significant. American Statistician, 60, 328–331.

MacCallum, R. C., Zhang, S., Preacher, K. J., & Rucker, D. D. (2002). On the practice of dichotomization of quantitative variables. Psychological Methods, 7, 19-40.

Nieuwenhuis, S., Forstmann, B. U., & Wagenmakers, E.-J. (2011). Erroneous analyses of interactions in neuroscience: A problem of significance. Nature Neuroscience, 14, 1105-1107.

Zou, G. Y. (2007). Toward using confidence intervals to compare correlations. Psychological Methods, 12, 399-413.

* Do I have any support for this position? Yes: anecodotal support (e.g., from editing or reviewing many dozens of papers) and some support from Nieuwenhuis et al. They found the error more prevalent in cellular and molecular neuroscience. ANOVA is core training in psychology and widely used in cognitive and behavioural neuroscience - and I'd argue that this reflects the influence of psychologists working in this area and of neuroscientists trained in and using similar methods.

** Do I have any support for this position? A little. It is easy to find basic psychology texts with ANOVA but without tests of differences in correlations being mentioned. It is rare to find tests of CIs of differences in published papers.

R: An introduction for psychologists

2011-01-13T13:50:00.000Z

Here are the slides from the Introduction to R session Danny Kaye and I ran at the BPS Mathematics, Statistics & Computing section CPS Workshop (13 December 2010, Nottingham Trent University).

Introduction to using R in research

2010-05-13T16:21:00.000+01:00

I was recently asked to give a talk to our graduate school annual conference. I offered several titles and the one they picked was Using R in research. I'm not sure if this was a good idea or not. The graduate school covers PhD students across three areas of the university: social sciences (including psychology), law and business. In addition the students tend to specialize in either qualitative or quantitive research methods, so I was talking to an audience who might know nothing about statistics or a great deal (e.g., several students have completed MSc courses in psychological research methods here or elsewhere).

My solution was try and explain the advantages of R relative to alternatives such as SPSS (probably the most common statistic package in the University). I also focussed a lot on graphical methods and simulation. It seemed to go quite well, but I worry that quite a few members of the audience were overwhelmed by large chunks of it.

I promised to put my slides on my blog - though I am not sure how useful they are to anybody who wasn't there. Without my commentary some (possibly most) of the slides won't make much sense. I spent a good deal of the time talking through exploratory plots of one data set (from Hayden, 2005). I use this example a lot in teaching and it involves a bit of class participating (guessing the origin of the data) - so I won't go into to detail here (lest I spoil it for future students), but you can google the original article if you are curious. I also spent some time on how R works (e.g., object types, assignment, basic modeling, plotting functions). My reasoning was that many of the audience have no familiarity with non-GUI interfaces in software and without explaining the basics of the interface they will not have the faintest clue how R works. For those with some familiarity (e.g., SPSS syntax) the examples were selected to show how powerful R can be for things like exploratory graphics.

Several students ask about resources for learning R. I mentioned some in earlier blog posts, but for psychologists Li and Baron's web resources are a good place to start. The other major resource is probably Quick R, but there are hundreds of other places to look online (depending on what stuff you need most).

Example of plotting a serial position curve in R

2010-03-03T20:14:00.018Z

A while ago I wrote a co-wrote chapter for an introductory psychology text book Essential Psychology: A Concise Introduction. This is a book edited and written by members of the department where I work. My contribution was the chapter on human memory (cunningly titled Memory).

I produced several plots for the chapter (some of which got cut due to severe space restrictions). One that stayed in was a serial position curve. For this plot I used data from Postman and Phillips (1965).

I feel particular proud of this plot because I was just beginning to use and learn R at the time (as opposed to dabbling) and because I had had a really hard time getting hold of the data. I first tried google, but had no joy (for some reason I thought someone would have put the raw data online, as it is a classic study - though maybe I just missed it). Then I searched for alternative data sets (as around that period there were quite a few similar studies). I was probably being too picky, but whatever the reason I had no luck.

It would have been trivial to make up fake data, but that didn't feel right. What I eventually did (and wished I'd done straight away) was print out the original figure and measure all the points by hand. I then entered these values into a spreadsheet and tweaked and remeasured until all the summary statistics matched those in the original paper to about one decimal place. This was a lot quicker than I had thought. I cheated slightly because I only needed data from the 20 word conditions (so I could leave out the 10 and 30 word conditions).

(I'm pretty sure I could have used computer software to capture the raw data from an image file, but I'd have had to find the software, learn how to use it and do all the checking anyway. For a single figure I'm reasonably sure measuring by hand would be faster.)

In re-plotting it I noticed a few things that I hadn't paid much attention to before. The main one was the authors report frequency of recalls for 18 participants with 6 lists each. This means all scores are out of 108 and I suspect lots of casual readers would (like me) assume they were percentages. For re-plotting I rescaled the data as percentages.

The plot itself just uses basic R functions. I'm writing about it because: i) I think it is a fairly clear illustration of how basic plot functions in R can produce what I think is a rather nice Figure. (The published version has been edited by the publisher, adding colour and making the style match figures in other chapters), ii) people may find it useful for teaching purposes. So please feel free to use and adapt the R code for non-commercial (e.g., teaching use).

First load the data from this .csv file (you will need to specify the path or change the working directory if the file is saved elsewhere).

pp65 <- read.csv("pp65.csv")

Then paste the following:

plot(pp65$SP, pch=NA, ylim=c(0,80), xlab= "Serial position", ylab= "Mean percentage recall", main = "Postman & Phillips (1965)", sub = '(20 word conditions only)')

points(pp65$C0, pch=19, col='black', cex=.7)

lines(pp65$C0, lty=3)

points(pp65$C15, pch=24, col='black', cex=.7)

lines(pp65$C15, lty=2)

points(pp65$C30, pch=22, col='black', cex=.7)

lines(pp65$C30, lty=5)

legend(3, 80, legend=c("No delay","15 second delay","30 second delay"), lty=c(3,2,5))

If you are new to R you can find out more about these plotting functions by using R help: ?par, ?plot, ?points and so on ...

References

Baguley, T., & Edmonds, A. J. (2010). Memory. In P. Banyard, M. N. O. Davies, C. Norman, & B. Winder (Eds.) Essential Psychology: A Concise Introduction (pp. 65-82). London: Sage.

Postman, L. & Philips, L. W. (1965). Short-term temporal changes in free recall. Quarterly Journal of Experimental Psychology, 17, 132-138.

Interaction plot from cell means

2010-02-25T00:23:00.001Z

I needed to produce a few a interaction plots for my book in R and, while the interaction.plot() function is useful it has a couple of drawbacks. First, the default output isn't very pretty. Second, it works from the raw data, whereas I often need plots from cell means. For teaching purposes it is quite common to produce plots without raw data (for hypothetical data or from published examples).

My first attempts at the plots involved setting them up element by element. Just going over some examples I decided to turn the basic plot (for a 2 x 2 ANOVA) into a simple function. Nothing fancy, just a regular interaction plot in black and white that I think is prettier than the SPSS, Excel or R defaults. At some point I may have a go turning it into a general I x J ANOVA plot (or maybe even add CIs, but I'll probably do that from raw data if I ever get round to it).

plot.2by2 <- function(A1B1,A1B2, A2B1, A2B2, group.names, legend = TRUE, leg.loc=NULL, factor.labels=c('Factor A', 'Factor B'), swap = FALSE, ylab= NULL, main = NULL){
group.means <- c(A1B1, A2B1, A1B2, A2B2)
if(missing(ylab)) ylab <- expression(italic(DV))
if(swap==TRUE) {
group.names <- list(group.names[[2]], group.names[[1]]) ; group.means <- c(A1B1, A1B2, A2B1, A2B2); factor.labels <- c(factor.labels[2], factor.labels[1])
}
plot(group.means, pch=NA, ylim=c(min(group.means)*.95, max(group.means)*1.025), xlim=c(0.8,2.2), ylab=ylab, xaxt='n', xlab=factor.labels[1], main=main)
points(group.means[1:2], pch = 21)
points(group.means[3:4], pch = 19)
axis(side = 1, at = c(1:2), labels = group.names[[1]])
lines(group.means[1:2], lwd = .6, lty = 2)
lines(group.means[3:4], lwd = .6)
if(missing(leg.loc)) leg.loc <- c(1,max(group.means))
if(legend ==TRUE) legend(leg.loc[1], leg.loc[2],legend = group.names[[2]], title = factor.labels[2], lty = c(3,1))
}

Call the function by entering the four cell means in conventional order: A1B1, A1B2 and so on where A1B1 is the mean of level 1 of factor A at level 1 of factor B. You also need a two item list containing text strings of the two level names of each factor. For instance:

lev.names <- list(c('A1', 'A2'), c('B1', 'B2'))
plot.2by2(5,15,10,20, lev.names)

You can swap the axes by adding the argument swap = TRUE:

plot.2by2(5,15,10,20, lev.names, swap = TRUE)

The default factor names are 'Factor A' and 'Factor B', but these are over-ridden in the call:

plot.2by2(5,15,10,20,lev.names, swap = TRUE, factor.labels= c('Factor 1','Factor 2'))

You can also change the y-axis label with ylab or add a main title with main. The legend can be dropped (legend = FALSE) if you don't want one or need it to be located outside the plot. To move the legend just specify coordinates with an argument such as leg.loc = c(1,10). You can also edit the source code directly. Here is an example with title and meaningful labels:

group.names <- list(c('placebo','drug'), c('male', 'female'))

plot.2by2(10,10,15,20, group.names, factor.labels=c('Drug', 'Sex'), swap = FALSE)

As this just uses basic plotting functions in R you can also manipulate the plot in other ways: adding lines with segments(), adding text with text() changing graphical parameters with par() and so on. Depending on your platform it is also easy to extract the plot as a .pdf or .jpg file. On a mac I save it as a .pdf file and open it in preview which allows me to save it as .png, .gif or whatever I need.