Power as a Ratio

For a “well-powered experiment,” the task is to get the ratio above at least 2.5 (for 80% power) and ideally to 3.3 (for 95% power).²

Here’s the idea:

Power equals , where is the standard normal CDF (as found in a standard table), is the standard error of the estimated treatment effect, and is the treatment effect.

This rule allows us to connect statistical power to the familiar table in the appendices of many statistics textbooks or the pnorm() function in R. But, more importantly, it drives home an important intuition—power is determined by the key ratio . To estimate power, we must make an informed assumption about the effect of interest and a good prediction of standard error of the estimate. Further, when we consider changes to the experimental design and the consequences for statistical power, it can be helpful to think about the numerator or the denominator of the key ratio : How do you make the effect as large as possible? And how do you make the estimate as precise as possible?

Kane’s List, and Relevance to the Ratio

Kane (2025) offers an excellent list of options to either increase the treatment effect or shrink the standard error.

He pitches the list as a way to make a compelling case for null results. However, the list is also helpful for making sure that we’ve thought through the ways to maximize power. It’s a wonderful paper well worth a careful read for folks who run experiments or read experimental work.

I’m going to borrow his list and connect the action items to statistical power.

Kane arranges his list into seven “alternative explanations” for null results, which we might think of as “potential power leakages.” I recreated a version of Kane’s table (Table 1 in his paper) that contains each action item to consider. I’ve kept Kane’s labeling for the various categories, but taken some liberties in collapsing some of the action items.³

The table below lists all of Kane’s categories and action items and notes the pathway through which this stop might increase statistical power. While Kane doesn’t explicitly pitch the list as “ways to increase” power, we can certainly borrow it for that purpose.

And for completeness, here is Kane’s original table:

Predicting the SE from pilot data

Here’s how I suggest we use pilot data to predict the standard error in the planned study:

The factor nudges the standard error from the pilot study in a conservative direction, since it might be an under-estimate of the actual standard error.² For the details, see this paper, but this conservative standard error estimate is approximately the upper bound of a 95% confidence interval for the standard error using the pilot data.

The data

In the {crdata} package on GitHub, you can find the the pilot data we collected leading up to the main study.

# download the {crdata} package from github
remotes::install_github("carlislerainey/crdata")

Now let’s load the pilot data. To focus on observations where the operation succeeds, we’re going to keep only the observations where the vignette describes a successful observation.

# load pilot data and keep only success condition
robbins2_pilot <- crdata::robbins_pilot |>
  subset(failure == "Success") |>
  glimpse()

Rows: 147
Columns: 5
$ cong_overall <dbl> 3, 1, -2, 0, -2, -1, 0, -1, 0, 0, 0, 2, -1, 3, -3, 0, 0, …
$ failure      <fct> Success, Success, Success, Success, Success, Success, Suc…
$ amplify      <fct> Ignore, Ignore, Ignore, Amplify, Ignore, Ignore, Ignore, …
$ pid7         <fct> Strong Democrat, Not very strong Republican, Strong Democ…
$ pid_strength <dbl> 3, 2, 3, 2, 2, 3, 2, 2, 2, 2, 2, 1, 0, 3, 3, 3, 3, 1, 1, …

cong_overall is the respondent’s approval of Congress’ actions on a seven-point scale and amplify indicates whether Congress amplified the whistleblower (i.e., criticized the president).

# t test
fit_pilot <- t.test(cong_overall ~ amplify, data = robbins2_pilot)

# create a table showing the observations per condition
table(robbins2_pilot$amplify)

 Ignore Amplify 
     70      77 

# sample size per condition
n_pilot <- mean(table(robbins2_pilot$amplify))
n_pilot

[1] 73.5

# get estimated standard error from pilot
se_hat_pilot <- fit_pilot$stderr
se_hat_pilot

[1] 0.2761011

n_planned <- 250

pred_se_cons <- sqrt(n_pilot/n_planned)*((sqrt(1/n_pilot) + 1)*se_hat_pilot)
pred_se_cons

[1] 0.1671691

# compute conservative minimum detectable effect
2.5*pred_se_cons

[1] 0.4179227

# compute power as a percent
1 - pnorm(1.64 - 0.35/pred_se_cons)

[1] 0.6749736

# without the conservative nudge
pred_se <- sqrt(n_pilot/n_planned)*se_hat_pilot
2.5*pred_se  # best guess of minimum detectable effect

[1] 0.3742673

1 - pnorm(1.64 - 0.35/pred_se)  # best guess of power

[1] 0.7573806

n_pilot*((2.5/0.35)*((sqrt(1/n_pilot) + 1)*se_hat_pilot))^2

[1] 356.4477

n_planned <- 367
pred_se_cons <- sqrt(n_pilot/n_planned)*((sqrt(1/n_pilot) + 1)*se_hat_pilot)
pred_se_cons

[1] 0.1379726

2.5*pred_se_cons  # conservative minimum detectable effect

[1] 0.3449315

1 - pnorm(1.64 - 0.35/pred_se_cons)  # conservative power

[1] 0.8150699

How did we do?

We ran the full study.⁴

robbins2_main <- crdata::robbins_main |>
  subset(failure == "Success") |>
  glimpse()

Rows: 735
Columns: 5
$ cong_overall <dbl> 2, -2, -1, -3, 0, -1, -2, -1, 1, 1, 0, -3, -2, 2, 2, -3, …
$ failure      <fct> Success, Success, Success, Success, Success, Success, Suc…
$ amplify      <fct> Ignore, Amplify, Amplify, Amplify, Ignore, Ignore, Amplif…
$ pid7         <fct> Not very strong Republican, Not very strong Republican, S…
$ pid_strength <dbl> 2, 2, 3, 3, 3, 2, 0, 2, 2, 1, 2, 0, 3, 1, 2, 3, 2, 3, 2, …

fit_main <- t.test(cong_overall ~ amplify, data = robbins2_main)
fit_main$stderr

[1] 0.1322618

1 - pnorm(1.64 - 0.35/fit_main$stderr)

[1] 0.8428563

As you can see, the pilot data gave us a good, slightly conservative prediction. We conservatively predicted a standard error of 0.138 in the planned study and we estimated a standard error of 0.132 after running the study. We conservatively predicted our power would be about 82% to detected an effect of 0.35 on the seven-point scale, but after running the study, it seems like we had about 84% power.

Pilot data should not usually be used to estimate the treatment effect

To compute statistical power, researchers need to make an assumption about the size of the treatment effect. It’s easy to feel lost without any guidance on what effects are reasonable to look for, so we might feel tempted to use a small pilot study to estimate the treatment effect and then use that estimate in our power analysis. This is a bad idea because the estimate of the treatment effect from a pilot study is too uncertain for a power analysis.² Leon, Davis, and Kraemer (2011) and Albers and Lakens (2018) discuss this problem in more detail.³

Pilot data can be used to predict the standard error

While pilot data might not be useful for estimating the treatment effect, pilot data are useful for estimating the standard error of the planned study. Given that power is a function of the ratio of the treatment effect and the standard error, it’s important to have a good prediction of the standard error. Further, the noisiness of this estimated standard error is predictable, so it’s easy to nudge the estimate slightly to obtain a conservative prediction.

In political science, it’s common to run pilot studies with, say, 100-200 respondents before a full-sized study of, say, 1,000 respondents. It can be very helpful to use these pilot data to confirm any preliminary power calculations.

Here are two helpful rules:

Note that the factor nudges the predicted standard error in a conservative direction. See this working paper for more details.

We can use the predicted standard error to find the minimum detectable effect (for 80% power) or the power (for a given treatment effect). Or we can use the pilot data to estimate the required sample size (for 80% power to detect a given treatment effect.)

# set the treatment effect, SE, and sample size
tau <- 1  # treatment effect
se  <- tau/(qnorm(0.95) + qnorm(0.80))  # standard error for 80% power
n_planned <- 500  # sample size per condition in planned full study

# calculate required standard deviation to yield 80% power given the above
sigma <- se*sqrt(2*n_planned)/2

res_list <- NULL # a container to collect results
for (i in 1:10000) {
      
  # simulate study
  y0 <- rnorm(2*n_planned, sd = sigma)
  y1 <- y0 + tau
  d <- sample(rep(0:1, length.out = 2*n_planned))
  y <- ifelse(d == 1, y1, y0)
  data <- data.frame(y, d)
  
  # fit model and get standard error and p-value
  fit <- lm(y ~ d, data = data)
  tau_hat <- as.numeric(coef(fit)["d"])
  se_hat <- as.numeric(sqrt(diag(sandwich::vcovHC(fit, type = "HC2")))["d"])
  p_value <- pnorm(tau_hat/se_hat, lower.tail = FALSE)

  # collect results
  res_list[[i]] <- data.frame(tau_hat, se_hat, p_value)
  }

# compute power (and monte carlo error)
res_list |>
  bind_rows() |>
  summarize(power = mean(p_value < 0.05), 
            mc_error = sqrt(power*(1 - power))/sqrt(n()), 
            lwr = power - 2*mc_error, 
            upr = power + 2*mc_error)

   power    mc_error       lwr       upr
1 0.8023 0.003982646 0.7943347 0.8102653

# sample size per condition in pilot study
n_pilot_values   <- c(10, 30, 60, 90, 150) 

res_list <- NULL  # a container to collect results
iter <- 1  # counter to index the collection
for (i in 1:10000) {
  for (j in 1:length(n_pilot_values)) {
    
    # set respondents per condition in the pilot study
    n_pilot <- n_pilot_values[j]
    
    # simulate pilot study
    y0 <- rnorm(2*n_pilot, sd = sigma)
    y1 <- y0 + tau
    d <- sample(rep(0:1, length.out = 2*n_pilot))
    y <- ifelse(d == 1, y1, y0)
    pilot_data <- data.frame(y, d)
    
    # fit model and get standard error
    fit_pilot <- lm(y ~ d, data = pilot_data)
    tau_hat <- as.numeric(coef(fit_pilot)["d"])
    pilot_se_hat <- as.numeric(sqrt(diag(sandwich::vcovHC(fit_pilot, type = "HC2")))["d"])
    
    # collect standard errors 
    res_list[[iter]] <- data.frame(pilot_se_hat, n_pilot)
    iter <- iter + 1 # update counter
  }
}

# combine collected results in a data frame
res <- bind_rows(res_list) |>
  glimpse()

Rows: 50,000
Columns: 2
$ pilot_se_hat <dbl> 2.7282069, 1.7541542, 1.1426683, 0.9763233, 0.7760267, 3.…
$ n_pilot      <dbl> 10, 30, 60, 90, 150, 10, 30, 60, 90, 150, 10, 30, 60, 90,…

ggplot(res, aes(x = pilot_se_hat)) + 
  geom_histogram() + 
  facet_wrap(vars(n_pilot)) + 
  geom_vline(xintercept = se)

ggplot(res, aes(x = sqrt(n_pilot/n_planned)*pilot_se_hat)) + 
  geom_histogram() + 
  facet_wrap(vars(n_pilot)) + 
  geom_vline(xintercept = se)

ggplot(res, aes(x = sqrt(n_pilot/n_planned)*(sqrt(1/n_pilot) + 1)*pilot_se_hat)) + 
  geom_histogram() + 
  facet_wrap(vars(n_pilot)) + 
  geom_vline(xintercept = se)

Reproducing Clark and Golder (2006)

Before computing their specific quantities of interest, let’s reproduce their regression model. Here’s their table that we’re trying to reproduce.

And here’s a reproduction of their estimates using the cg2006 data from the {crdata} package on GitHub.¹

# load packages
library(tidyverse)
library(sandwich)
library(modelsummary)

# install my data packages from github
devtools::install_github("carlislerainey/crdata")  # only updates if newer version available

# load clark and golder's data set
cg <- crdata::cg2006

# reproduce their estimates
f <- enep ~ eneg*log(average_magnitude) + eneg*upper_tier + en_pres*proximity
fit <- lm(f, data = cg)

# regression table
modelsummary(fit, 
             vcov = ~ country, # cluster-robust SE; multiple observations per country
             fmt = 2, 
             shape = term ~ model + statistic)

Success!

They use averge_magnitude to measure the permissiveness of the electoral system and eneg to measure social heterogeneity.

Using comparisons() to compute the effects

Now let’s compute the two quantities of interest. Clark and Golder argue for two negligible effects, which I make really concrete below.

• Hypothesis 1 Increasing the effective number of ethnic groups from the 10th percentile (1.06) to the 90th percentile (2.48) will not lead to a substantively meaningful change in the effective number of political parties when the district magnitude is one.
• Hypothesis 2 Increasing the district magnitude from one to seven will not lead to a substantively meaningful change in the effective number of political parties when the effective number of ethnic groups is one.

And comparing the U.S. and the U.K., I argue that the smallest substantively interesting effect is 0.62. In Rainey (2014), I made the plot below. I want to reproduce it with {marginaleffects}.

These differences (and the 90% CIs) are really easy to compute using {marginaleffects}!²

# load packages
library(marginaleffects)

# the smallest substantively interesting effect
m <- 0.62

# a data frame setting the values of the "other" variables
X_c <- data.frame(
  eneg = 1.06,  # low value
  average_magnitude = 1,  # low value
  upper_tier = 0,
  en_pres = 0, 
  proximity = 0
)

# compute the comparison for eneg and average magnitude
c <- comparisons(fit,
            vcov = ~ country,
            newdata = X_c, 
            variables = list("eneg" = c(1.06, 2.48),         # low to high value
                             "average_magnitude" = c(1, 7)), # low to high value
            conf_level = 0.90)

This c outputted from comparisons() is a data frame.

# default print method
print(c)

              Term    Contrast Estimate Std. Error     z Pr(>|z|)    S  5.0 %
 average_magnitude 7 - 1          0.696      0.175 3.973   <0.001 13.8  0.408
 eneg              2.48 - 1.06    0.158      0.204 0.779    0.436  1.2 -0.176
 95.0 %
  0.984
  0.493

Type:  response 

# quick look at the bare data frame
glimpse(c)

Rows: 2
Columns: 19
$ rowid             <int> 1, 1
$ term              <chr> "average_magnitude", "eneg"
$ contrast          <chr> "7 - 1", "2.48 - 1.06"
$ estimate          <dbl> 0.6956014, 0.1584771
$ std.error         <dbl> 0.1750741, 0.2035052
$ statistic         <dbl> 3.9731830, 0.7787372
$ p.value           <dbl> 7.091852e-05, 4.361345e-01
$ s.value           <dbl> 13.783478, 1.197155
$ conf.low          <dbl> 0.4076302, -0.1762592
$ conf.high         <dbl> 0.9835727, 0.4932134
$ predicted_lo      <dbl> 3.034008, 3.034008
$ predicted_hi      <dbl> 3.729609, 3.192485
$ predicted         <dbl> 3.034008, 3.034008
$ eneg              <dbl> 1.06, 1.06
$ average_magnitude <dbl> 1, 1
$ upper_tier        <dbl> 0, 0
$ en_pres           <dbl> 0, 0
$ proximity         <dbl> 0, 0
$ enep              <dbl> 5.75, 5.75

Now we can just plot the 90% CIs with ggplot() and check whether the entire interval falls inside the bounds.

# bind the comparisons together and plot
ggplot(c, aes(x = estimate,
                 xmin = conf.low,
                 xmax = conf.high, 
                 y = term)) + 
  geom_vline(xintercept = c(-m, m), linetype = "dashed") + 
  geom_errorbarh() + 
  geom_point() 

In this case, we conclude that social heterogeneity (eneg) has a negligible effect because the 90% CI only contains substantively negligible values. However, the 90% CI for district magnitude (average_magnitude) contains substantively negligible and meaningful values, so we cannot reject the null hypothesis of a meaningful effect.

Final thoughts

1. {marginaleffects} is a great package (Arel-Bundock 2024). I think it’s the first package in which the syntax matches the way I think about computing quantities of interest. That said, this is just my first try at it. But I’m very impressed so far.
2. The {marginaleffects} book Model to Meaning has a whole chapter on equivalence tests. My only caution is that there is a mismatch between 95% confidence intervals and equivalence tests. By default, {marginaleffects} reports a 95% CI, even when producing a p-value for an equivalence test. However, the 90% confidence interval correspondents to a size-5% equivalence test. So if you’re using {marginaleffects} to do equivalence tests, I recommend setting conf_level = 0.90.³
3. For a more recent example, Jares and Malhotra (2024) discusses negligible effects and hypothesis tests in a way that I find clear and compelling. It’s an excellent model to follow. See pp. 12-13. They “show that improved compensation outcomes had negligible impacts on Republican farmers’ midterm turnout and campaign contributions, even though such variation in benefits significantly affected farmers’ propensity to view the intervention as helpful.”

The Evidence

Robert Boice studied academic productivity carefully. A couple of his studies provide some evidence for my strategy to write every day.

First, he assessed how early-career academics spend their time. The figure below shows the results. Notice that these faculty spend more time in committee meetings (2 hrs.) than writing (1.5 hours).

Second, Boice conducted an experiment to assess the effect of writing strategies.

Boice randomly divided 27 academics into three groups:²

• The control group agreed to defer all but the most urgent writing for ten weeks.
• The spontaneous group agreed write when they felt like it.
• The contingency group agreed to donate to an anti-charity if they failed to write every day.

The figure below shows that regular writing routine increase production of both pages and ideas. Notice that the spontaneous writers barely produced more ideas and pages than the group trying to avoid writing.

I find these results compelling, but note that Helen Sword urges some caution.

How I Do It

Everyone is different, and my own approach has evolved over time. Here are the key ingredients (for me):

1. Write for two hours at a regular time. Consistency is key.³
2. Avoid writing outside this window. Set your window so that your window is “enough.”
3. Take breaks. I take long breaks from writing. But these are intentional and planned.⁴
4. Family permitting, I think it’s helpful to spend a little while pushing the projects forward on the weekends, just to keep the momentum up.⁵

I admit that I deviate from the strategy above (and not always intentionally). But I’ve been at this long enough to know that a regular routine works really well for me.

Starting Point: The Sampling Distribution

When you run one experiment, you realize one of many possible patterns of randomization. This particular realization produces a single estimate of the treatment effect from a distribution of possible estimates. The distribution of possibilities is called a “sampling distribution.”

Thus, we can think of the estimate as a random variable and its distribution as the sampling distribution. The sampling distribution is key to everything I do in this post, so let’s spend some time with it.

Let’s imagine that we did the exact same study 50 times. Let’s say that we computed a difference-in-means in dollars ($) donated.¹ I refer to this difference-in-means as the estimated treatment effect. It is the estimate of the average treatment effect (in $). This estimate will vary across the many possible patterns of randomization because each pattern puts different respondents in the treatment and control group.

We can visualize this with ggnaminate. We can imagine each iteration of the study as producing a particular estimate. We continue to repeat the study and collect the estimates. Eventually, we can produce a histogram from this collection of estimates. This histogram represents the sampling distribution and is fundamental to the calculations that follow. The figure belows shows how we might collect the points into a histogram.

library(tidyverse)
library(gganimate)
library(magick)

# gif pars
duration <- 24 # must be even
fps <- 25
nframes <- duration*fps
scale <- 2.5
width <- 8
height <- 6
res <- 125

# study parameters
true_effect <- 1
se <- 0.4

# number of times to repeat the study
n_studies <- 50 # nframes

# create a data frame of confidence intervals
ests <- tibble(study_id = 1:n_studies, 
               est = c(rnorm(n_studies, true_effect, se))) %>%
  mutate(reject_null = ifelse(est - 1.64*se > 0, "Yes", "No"))

# add two things to the data frame of confidence intervals
# 1. an initial row with study_id = 1 and est = NA so that 
#    the plot starts empty (gganimate would start with the 
#    first observation in place otherwise).
# 2. a group variable that defines the row. This is the same
#    as the study_id, except the dummy row from (1) and the 
#    actual first row have different groups.
animate_data <- bind_rows(
  tibble(study_id = 1, est = NA),  # study_id = 1, est = NA
  ests                             # actual cis
) %>%
  mutate(group = 1:n()) 

split_animate_data <- animate_data %>%        # group (row index)
  split(.$group) %>%
  accumulate(~ bind_rows(.x, .y)) %>% 
  bind_rows(.id = "frame") %>% 
  mutate(frame = as.integer(frame))


se_lines <- tribble(
  ~se_, ~label, ~chance, ~ch_loc_,  # trailing _ means not rescaeld to study se
  0, "True Effect", NA, NA,
  1, "+1 SE", scales::percent(pnorm(1) - pnorm(0), accuracy = 1), 0.5,
  2, "+2 SE", scales::percent(pnorm(2) - pnorm(1), accuracy = 1), 1.5,
  3, "+3 SE", scales::percent(pnorm(3) - pnorm(2), accuracy = 1), 2.5,
  -1, "-1 SE", scales::percent(pnorm(0) - pnorm(-1), accuracy = 1), -0.5,
  -2, "-2 SE", scales::percent(pnorm(-1) - pnorm(-2), accuracy = 1), -1.5,
  -3, "-3 SE", scales::percent(pnorm(-2) - pnorm(-3), accuracy = 1), -2.5,
) %>%
  mutate(ch_loc = ch_loc_*se + true_effect,
         se = se_*se + true_effect)

# start with a ggplot
gg1 <- ggplot(animate_data, aes(x = est, 
                               y = study_id,
                               group = group)) + 
  geom_vline(data = se_lines, aes(xintercept = se, 
                                  color = -dnorm(se_)), linetype = "dashed") + 
  geom_label(data = se_lines, aes(x = se, y = n_studies + 2, label = label, group = NULL, color = -dnorm(se_))) + 
  geom_text(data = se_lines, aes(x = ch_loc, y = 4, label = chance, group = NULL)) + 
  geom_point(aes(color = -dnorm((est- true_effect)/se)),
             size = 3) + 
  geom_rug(sides = "b", 
           aes(x = est, 
               color = -dnorm((est- true_effect)/se)), 
           alpha = 0.5, 
           length = unit(0.025, "npc")) + 
  theme_bw() + 
  theme(panel.grid.minor.y = element_blank()) + 
  labs(x = "Estimate of Effect",
       y = "Study Number") +
  theme(legend.position = "none")

# add dyamics to the plot
gg1_anim <- gg1 +  
  transition_states(states = group) + 
  # how points enter
  enter_drift(y_mod = 10) +
  enter_grow() +
  enter_fade() + 
  # how points exit/remain
  exit_fade(alpha = 0.5) +
  exit_shrink(size = 1) + 
  shadow_mark(alpha = 0.5) 

gg1_gif<- animate(gg1_anim, nframes = nframes, duration = duration, width = width, height = height, units = "in", res = res)
anim_save("gg1.gif")
gg1_mgif <- image_read("gg1.gif")


## plot 2: histogram
# start with a ggplot
gg2 <- ggplot(split_animate_data, aes(x = est, group = frame)) + 
  geom_histogram(binwidth = se, boundary = true_effect, fill = "grey") + 
  geom_vline(data = se_lines, aes(xintercept = se, 
                                  color = -dnorm(se_)), linetype = "dashed") + 
  geom_label(data = se_lines, aes(x = se, y = Inf, label = label, group = NULL, color = -dnorm(se_)), vjust = 1.5) + 
  geom_label(data = se_lines, aes(x = ch_loc, y = 0, label = chance, group = NULL), vjust = -1) + 
  #geom_density(linewidth = 2) + 
  geom_rug(sides = "b", 
           aes(x = est, 
               color = -dnorm((est- true_effect)/se)), 
           alpha = 0.5, 
           length = unit(0.025, "npc")) + 
  theme_bw() + 
  labs(x = "Estimate of Effect",
       y = "Count") + 
  theme(legend.position = "none")

# add dyamics to the plot
gg2_anim <- gg2 +  
  transition_states(states = frame)

gg2_gif<- animate(gg2_anim, nframes = nframes, duration = duration, width = width, height = height, units = "in", res = res)
anim_save("gg2.gif")
gg2_mgif <- image_read("gg2.gif")



new_gif <- image_append(c(gg1_mgif[1], gg2_mgif[1]), stack = FALSE)
for(i in 2:nframes){
  combined_gif <- image_append(c(gg1_mgif[i], gg2_mgif[i]), stack = FALSE)
  new_gif <- c(new_gif, combined_gif)
}
new_gif

Before the study, we can predict two features of this sampling distribution.

1. First, it will usually have a bell-shaped, normal distribution.
2. Second, we can predict the standard deviation of this distribution with good accuracy before conducting a single study and excellent accuracy after just one study. We call the standard deviation of the sampling distribution the standard error or SE.

For our purposes, then, we can describe the sampling distribution as normally distributed with an assumed mean and SD. The mean is the assumed true effect and the SD is the well-predicted SE.

This is the key claim: In order to build power into your experiment, you must build certain properties into the sampling distribution.

The design of your experiment will not affect the normality of the sampling distribution, but it will change the true effect and the SE. Changing the true effect and the SE will change the power.

Thinking about True Effect and Standard Error as Targets

I’m leaving aside how to predict the standard error of the experiment or choose the true effect. This post is about the target standard error and true effect.

Predicting the standard error is a mechanical process mixed with a little guesswork.² Choose a true effect is a mostly substantive decision.³

But instead of talking about a target power, I like to talk about a target standard error (given a true effect) or a target true effect (given a standard error)—power is just not a very intuitive quantity.

To understand how the true effect and the SE relate to power, we need to introduce the confidence interval.

Testing with Confidence Intervals

I like to use 90% confidence intervals to test hypotheses (see Rainey 2014 and Rainey 2015). In short, 90% confidence intervals correspond to one-tailed tests with size 0.05 and equivalence tests with size 0.05.] The formula for a 90% CI is . That is, we put “arms” around our estimate—one to the left and another to the right. Each arm is 1.64 standard errors long.

I’ll assume we have a one-sided research hypothesis that suggests a positive effect. If the lower bound () is less than zero, then we fail to reject the null hypothesis. If this lower bound is greater than zero, then we reject the null hypothesis.⁴

This focus on testing rather than estimation changes the nature of the sampling distribution. Rather than an estimate along a continuous range, we get a binary outcome: either (1) reject the null hypothesis or (2) fail to reject the null hypothesis.

But the sampling distribution of estimates and the associated outcomes of tests are closely related. In particular, the logic of the test implies that *if the estimate falls less than 1.64 SEs above zero, we cannot reject the null.

We can reconstruct the figure above using this logic. Rather than plot the points continuously along the x-axis, we can color the points (and now error bars) according to whether the lower bound falls above zero or not. And we can use a bar plot showing the number of rejections and non-rejections.

library(tidyverse)
library(gganimate)
library(magick)

# gif pars
duration <- 24 # must be even
fps <- 25
nframes <- duration*fps
scale <- 2.5
width <- 8
height <- 6
res <- 125

# study parameters
true_effect <- 1
se <- 0.4

# number of times to repeat the study
n_studies <- 50 # nframes

# create a data frame of confidence intervals
ests <- tibble(study_id = 1:n_studies, 
               est = c(rnorm(n_studies, true_effect, se))) %>%
  mutate(reject_null = ifelse(est - 1.64*se > 0, "Yes", "No"),
         lwr = est - 1.64*se,
         upr = est + 1.64*se)

# add two things to the data frame of confidence intervals
# 1. an initial row with study_id = 1 and est = NA so that 
#    the plot starts empty (gganimate would start with the 
#    first observation in place otherwise).
# 2. a group variable that defines the row. This is the same
#    as the study_id, except the dummy row from (1) and the 
#    actual first row have different groups.
animate_data <- bind_rows(
  tibble(study_id = 1, est = NA),  # study_id = 1, est = NA
  ests                             # actual cis
) %>%
  mutate(group = 1:n()) 

split_animate_data <- animate_data %>%        # group (row index)
  split(.$group) %>%
  accumulate(~ bind_rows(.x, .y)) %>% 
  bind_rows(.id = "frame") %>% 
  mutate(frame = as.integer(frame))


se_lines <- tribble(
  ~se_, ~label, ~chance, ~ch_loc_,  # trailing _ means not rescaeld to study se
  0, "True Effect", NA, NA,
  1, "+1 SE", scales::percent(pnorm(1) - pnorm(0), accuracy = 1), 0.5,
  2, "+2 SE", scales::percent(pnorm(2) - pnorm(1), accuracy = 1), 1.5,
  3, "+3 SE", scales::percent(pnorm(3) - pnorm(2), accuracy = 1), 2.5,
  -1, "-1 SE", scales::percent(pnorm(0) - pnorm(-1), accuracy = 1), -0.5,
  -2, "-2 SE", scales::percent(pnorm(-1) - pnorm(-2), accuracy = 1), -1.5,
  -3, "-3 SE", scales::percent(pnorm(-2) - pnorm(-3), accuracy = 1), -2.5,
) %>%
  mutate(ch_loc = ch_loc_*se + true_effect,
         se = se_*se + true_effect)

# start with a ggplot
gg1 <- ggplot(animate_data, aes(x = est, 
                                y = study_id,
                                group = group)) + 
  geom_vline(xintercept = 1.64*se) + 
  annotate("label", x = 1.64*se, y = 5, label = "1.64 SEs") + 
  geom_errorbarh(height = 0, aes(xmin = lwr, xmax = upr, color = reject_null)) + 
  geom_point(aes(color = reject_null),
             size = 3) + 
  geom_rug(sides = "b", 
           aes(x = est, 
               color = reject_null), 
           alpha = 0.5, 
           length = unit(0.025, "npc")) + 
  theme_bw() + 
  theme(panel.grid.minor.y = element_blank()) + 
  labs(x = "Estimate of Effect",
       y = "Study Number") +
  theme(legend.position = "none") + 
  scale_color_manual(values = c("Yes" = "#1b9e77", "No" = "#d95f02"))

# add dyamics to the plot
gg1_anim <- gg1 +  
  transition_states(states = group) + 
  # how points enter
  enter_drift(y_mod = 10) +
  enter_grow() +
  enter_fade() + 
  # how points exit/remain
  exit_fade(alpha = 0.5) +
  exit_shrink(size = 1) + 
  shadow_mark(alpha = 0.5) 

gg1_gif<- animate(gg1_anim, nframes = nframes, duration = duration, width = width, height = height, units = "in", res = res)
anim_save("gg1.gif")
gg1_mgif <- image_read("gg1.gif")


## plot 2: histogram
# start with a ggplot
gg2 <- ggplot(split_animate_data, aes(x = reject_null, fill = reject_null),  na.rm = TRUE) + 
  geom_bar(na.rm = TRUE) + 
  theme_bw() + 
  labs(x = "Reject Null",
       y = "Count") + 
  theme(legend.position = "none") + 
  scale_x_discrete(na.translate = FALSE) + 
  scale_fill_manual(values = c("Yes" = "#1b9e77", "No" = "#d95f02"))

# add dyamics to the plot
gg2_anim <- gg2 +  
  transition_states(states = frame)

gg2_gif<- animate(gg2_anim, nframes = nframes, duration = duration, width = width, height = height, units = "in", res = res)
anim_save("gg2.gif")
gg2_mgif <- image_read("gg2.gif")



new_gif <- image_append(c(gg1_mgif[1], gg2_mgif[1]), stack = FALSE)
for(i in 2:nframes){
  combined_gif <- image_append(c(gg1_mgif[i], gg2_mgif[i]), stack = FALSE)
  new_gif <- c(new_gif, combined_gif)
}
new_gif

The Key Ratio

The key to building statistical power into your experiment is to get “almost all” of the sampling distribution above 1.64 standard errors above zero. The portion of the sampling distribution that falls below 1.64 standard errors above zero does not allow the researcher to reject the null.

library(tidyverse)

# study parameters
true_effect <- 1
se <- 0.4


se_lines <- tribble(
  ~se_, ~label, ~chance, ~ch_loc_,  # trailing _ means not rescaeld to study se
  0, "True Effect", NA, NA,
  1, "+1 SE", scales::percent(pnorm(1) - pnorm(0), accuracy = 1), 0.5,
  2, "+2 SE", scales::percent(pnorm(2) - pnorm(1), accuracy = 1), 1.5,
  3, "+3 SE", scales::percent(pnorm(3) - pnorm(2), accuracy = 1), 2.5,
  -1, "-1 SE", scales::percent(pnorm(0) - pnorm(-1), accuracy = 1), -0.5,
  -2, "-2 SE", scales::percent(pnorm(-1) - pnorm(-2), accuracy = 1), -1.5,
  -3, "-3 SE", scales::percent(pnorm(-2) - pnorm(-3), accuracy = 1), -2.5,
) %>%
  mutate(ch_loc = ch_loc_*se + true_effect,
         se = se_*se + true_effect) 

x <- rnorm(500000, mean = true_effect, sd = se)
df <- data.frame(x)

bg_alpha <- 0.3
ggplot() + 
  geom_histogram(data = df, 
                 aes(x = x, y = after_stat(density)), binwidth = se, boundary = true_effect, fill = "grey", alpha = bg_alpha) + 
  geom_vline(data = se_lines, aes(xintercept = se, 
                                  color = -dnorm(se_)), linetype = "dashed", alpha = bg_alpha) + 
  geom_label(data = se_lines, aes(x = se, y = Inf, label = label, group = NULL), vjust = 1.5, color = alpha('black', bg_alpha)) + 
  geom_label(data = se_lines, aes(x = ch_loc, y = 0, label = chance, group = NULL), vjust = -1, color = alpha('black', bg_alpha)) + 
  geom_vline(xintercept = 0) + 
  geom_function(fun = dnorm, args = list(mean = true_effect, sd = se), size = 1) + 
  theme_bw() + 
  labs(x = "Estimate of Effect",
       y = "Density") + 
  geom_area(data = tibble(x = seq(1.64*se, 3*se + true_effect, by = 0.1)), aes(x = x), 
            stat = "function", fun = dnorm, args = list(mean = true_effect, sd = se),
            fill = "#d95f02", alpha = 0.1, xlim = c(1.64*se, 3**se + true_effect)) + 
  annotate("label", x = 1.3, y = .1, label = "fraction rejected", color = "#d95f02", size = 6) +
  annotate("segment", x = 1.64*se, xend = 1.64*se, y = 0, yend = dnorm(1.64*se, mean = true_effect, sd = se), color = "#1b9e77", size = 1) + 
  annotate("label", x = 1.64*se, y = dnorm(1.64*se, mean = true_effect, sd = se)/2, label = "1.64 SEs above zero", color = "#1b9e77") + 
  annotate("segment", x = 0, xend = 1.64*se, 
           y = dnorm(1.64*se, mean = true_effect, sd = se), 
           yend = dnorm(1.64*se, mean = true_effect, sd = se), 
           color = "#7570b3", size = 1) + 
  annotate("label", x = 0.5*1.64*se, y = dnorm(1.64*se, mean = true_effect, sd = se), label = "width of 90% CI", color = "#7570b3") + 
  theme(legend.position = "none") + 
  xlim(-1, 3)

You’ll remember that “almost all” of the normal distribution falls within two standard errors of its mean. So as a starting point, let’s use this rule: get the sampling distribution two standard errors above 1.64 standard errors above zero. We can just add 1.64 and 2 together to get a sampling distribution 3.64 standard errors above zero. That is, we need . If the true effect is larger than 3.64 standard errors, then the confidence interval will “rarely” overlap zero (about 2% of the time).

library(tidyverse)

# study parameters
true_effect <- 1
se <- 1/3.84

x <- rnorm(500000, mean = true_effect, sd = se)
df <- data.frame(x)

ggplot() + 
  geom_histogram(data = df, 
                 aes(x = x, y = after_stat(density)), binwidth = se, boundary = true_effect, fill = "grey", alpha = bg_alpha) + 
  geom_vline(data = se_lines, aes(xintercept = se, 
                                  color = -dnorm(se_)), linetype = "dashed", alpha = bg_alpha) + 
  geom_label(data = se_lines, aes(x = se, y = Inf, label = label, group = NULL), vjust = 1.5, color = alpha('black', bg_alpha)) + 
  geom_label(data = se_lines, aes(x = ch_loc, y = 0, label = chance, group = NULL), vjust = -1, color = alpha('black', bg_alpha)) + 
  geom_vline(xintercept = 0) + 
  geom_function(fun = dnorm, args = list(mean = true_effect, sd = se), size = 1) + 
  theme_bw() + 
  labs(x = "Estimate of Effect",
       y = "Density") + 
  geom_area(data = tibble(x = seq(1.64*se, 3*se + true_effect, by = 0.1)), aes(x = x), 
            stat = "function", fun = dnorm, args = list(mean = true_effect, sd = se),
            fill = "#d95f02", alpha = 0.1, xlim = c(1.64*se, 3**se + true_effect)) + 
  #annotate("label", x = 1.3, y = .1, label = "fraction rejected", color = "#d95f02", size = 6) +
  annotate("segment", x = true_effect, xend = true_effect, y = 0, yend = dnorm(true_effect, mean = true_effect, sd = se), color = "#d95f02", size = 1) + 
  annotate("label", x = true_effect, y = dnorm(true_effect, mean = true_effect, sd = se)/2, label = "true effect", color = "#d95f02") +   
  annotate("segment", x = 1.64*se, xend = 1.64*se, y = 0, yend = .75, color = "#1b9e77", size = 1) + 
  annotate("label", x = 1.64*se, y = .75, label = "1.64 SEs above zero", color = "#1b9e77") + 
  annotate("segment", x = 0, xend = 1.64*se, 
           y = .125, 
           yend = .125, 
           color = "#7570b3", size = 1,
           lineend = "round", linejoin = "round", arrow = arrow(length = unit(0.1, "inches"), ends = "both")) + 
  annotate("label", x = 0.5*1.64*se, y = .125, label = "1.64 SEs", color = "#7570b3", size = 4) + 
  annotate("segment", x = true_effect, xend = 1.64*se, 
           y = .125, 
           yend = .125, 
           color = "black", size = 1,
           lineend = "round", linejoin = "round", arrow = arrow(length = unit(0.1, "inches"), ends = "both")) + 
  annotate("label", x = 1.64*se + (true_effect - 1.64*se)/2, y = .125
  
  , label = "ideally 2 SEs", color = "black", size = 4) + 
  annotate("segment", x = true_effect, xend = 0, 
           y = 1, yend = 1, 
           color = "black", size = 1,
           lineend = "round", linejoin = "round", arrow = arrow(length = unit(0.1, "inches"), ends = "both")) + 
  annotate("label", x = true_effect/2, y = 1, label = "ideally 3.64 SEs", color = "black", size = 4) + 
  theme(legend.position = "none") + 
  xlim(-0.1, 2.0)

The Resulting Guidelines

There are a few ways to write the ratio:

These are all the same goals. I prefer the first, but they are all equivalent targets for power.

I like this ratio because it points to strategies to increase power. You can do two things:

1. Increase the true effect.
2. Decrease the standard error.

This ratio gets you thinking not what your power is, but how to increase it. Again, power isn’t a task for the experimenter, it’s the task.⁵ Kane (2023)⁶ suggests several ways researchers can increase the true effect or decrease the standard error.

I’ll highlight a few examples here. To increase the treatment effect, you can:

1. Increase attentiveness.
2. Ensure that respondents are not pre-treated.
3. Write strong treatments or “hit them between the eyes” (Kuklinski et al. 2000).

To decrease the standard error, you can:

1. Increase the sample size.
2. Use multiple measurements for the outcome.
3. Use a pre-post design (Clifford, Sheagley, and Piston 2022).

Implication

If you get the ratio to 3.64, then you’ll have 98%. This is much higher than the cutoff of 80% I hear suggested most often. If you want 80% power, rather than 98% power, you can change the 3.64 to 2.48.⁷ But notice that 3.64 and 2.48 are not very different. This has an important implication.

A researcher can lower the risk of a failed study from about 1 in 5 (80% power) to about 1 in 50 (98% power) by increasing the ratio from 2.48 to 3.64. This requires increasing the treatment effect by about 50% or shrinking the standard error by about 33%. Stated differently, if you are careless and let your treatment effect fall by 33% or let your standard error increase by 50%, then your risk of a failed study increases 10-fold! Small differences can matter a lot.

The implication is this: when you are on the cusp of a well-powered study (about 80% power), then small increases in the ratio have a disproportionate impact on your risk of a failed study.

Background

In this post, I address confidence intervals that are nestled right up against zero.¹ These intervals indicate that an estimate is “barely” significant. I want to be clear: “barely significant” is still significant, so you should still reject the null hypothesis.²

But I want to address a feeling that can come along with a confidence interval nestled right up against zero. A feeling of victory. It seems like a perfectly designed study. You rejected the null and collected just enough data to do it.

But instead, it should feel like a near-miss. Like an accident narrowly avoided. A confidence interval nestled right up against zero indicates that one of two things has happened: either you were (1) unlucky or (2) under-powered.

Because “unlucky” is always a possibility, we can’t learn much from a particular confidence interval, but we can learn a lot from a literature. A literature with well-powered studies produces confidence intervals that often fall far from zero. A well-powered literature does not produce confidence intervals that consistently nestle up against zero. Under-powered studies, though, do tend to produce confidence intervals that nestle right up against zero.

A Simulation

I’m going to explore the behavior of confidence intervals with a little simulation. In this simulation, I’m going to assert a standard error rather than create the standard error endogenously through sample size, etc. I use a true effect size of 0.5 and standard errors of 0.5, 0.3, 0.2, and 0.15 to create studies with 25%, 50%, 80%, and 95% power, respectively.³ I think of 80% as “minimally-powered” and 95% as “well-powered.”

I’m using a one-sided test (hypothesizing a positive effect), so I’ll use 90% confidence intervals with arms that are 1.64 standard errors wide. Let’s simulate some estimates from each of our four studies and compute their confidence intervals. I simulate 5,000 confidence intervals to explore below.

# load packages
library(tidyverse)

# create a parameter for the true effect
true_effect <- 0.5 # just assumed by me

# create a data frame of standard errors (with approximate power)
se_df <- tribble(
  ~se,    ~pwr,
  0.5,    "about 25% power",
  0.3,    "about 50% power",
  0.2,    "about 80% power",
  0.15,   "about 95% power"
)

# create function to simulate estimates for each standard error
simulate_estimates <- function(se, pwr) {
  tibble(
    est = rnorm(n_cis, mean = true_effect, sd = se),
    se = se,
    pwr = pwr
  )
}

# simulate the estimates, compute the confidence intervals, and wrangle
n_cis <- 5000  # the number of cis to create
ci_df <- se_df %>% 
  # simulate estimates
  pmap_dfr(simulate_estimates) %>%
  # compute confidence intervals
  mutate(lwr = est - 1.64*se, 
         upr = est + 1.64*se) %>%
  # summarize the location of the confidence interval
  mutate(result = case_when(lwr < 0 ~ "Not significant",
                            lwr < se ~ "Nestled against zero",
                            lwr >= se~ "Not nestled against zero"))

Now let’s quickly confirm my power calculations by computing the proportion of confidence intervals to the right of zero. These are about right. In a later post, I’ll describe how I think about computing these quantities.

# confirm power calculations
ci_df %>%
  group_by(se, pwr) %>%
  summarize(sim_pwr = 1 - mean(result == "Not significant"),
            sim_pwr = scales::percent(sim_pwr, accuracy = 1)) %>%
    select(SE = se, 
         Power = pwr,
         `Percent Significant` = sim_pwr) %>%
  kableExtra::kable(format = "markdown")

What Do Confidence Intervals from Well-Powered Studies Look Like?

Now let’s see what these confidence intervals look like. 5,000 is too many to plot, so I sample 25. But I applied the statistical significance filter first. This mimics the publication process and makes the plots a little easier to compare. My argument doesn’t depend on this filter, though.

I plotted these 100 intervals below⁴

There are three important vertical lines in these plots.

1. The solid line indicates zero. All confidence intervals are above zero because I applied the significance filter.
2. The dotted line indicates one standard error above zero. This varies across panels because the standard error varies across panels.
3. The dashed line indicates the true effect of 0.5. Because I applied the significance filter, the lower-powered studies are consistently over-estimating the true effect.

The intervals are green when the lower bound of the 90% confidence interval falls within one standard error of zero—that’s my definition of “nestled up against zero.” The intervals are orange when the lower bound falls further than one standard error above zero.

Notice how low-powered studies tend to nestle their confidence intervals right up against zero. Almost all of the confidence intervals from the study with 25% power are nestled right up against zero. Very few of the confidence intervals from the study with 95% power are nestled up against zero.

Again, you should apply this standard to a literature. You should not apply this standard to a particular study because even well-powered studies sometimes produce confidence intervals that nestle up against zero. But when you start to see confidence intervals consistently falling close to zero, you should start to assume that the literature uses under-powered studies and that the estimates in that literature are inflated due to Type M errors (Gelman and Stern 2014).

gg_df <- ci_df %>%
  filter(lwr > 0) %>% # apply significance filter 
  # sample 25 intervals (from those that are significant)
  group_by(se, pwr) %>%
  sample_n(25) %>%
  # create id (ordered by estimate value)
  group_by(se, pwr) %>%
  arrange(est) %>%
  mutate(ci_id = 1:n())
  
ggplot(gg_df, aes(x = est, xmin = lwr, xmax = upr, y = ci_id,
                    color = result)) + 
  facet_wrap(vars(pwr), ncol = 1, scales = "free_x") + 
  geom_vline(data = se_df, aes(xintercept = se), linetype = "dotted") +
  geom_vline(xintercept = 0) + 
  geom_vline(xintercept = true_effect, linetype = "dashed") + 
  geom_errorbarh(height = 0) + 
  geom_point() + 
  scale_color_brewer(type = "qual", palette = 2) + 
  theme_bw() + 
  labs(x = "Estimate and 90% CI",
       y = NULL,
       color = "Result")

Background

There’s been some really good work lately on statistical power. I’ll point you to two really great papers.

1. Arel-Bundock, Vincent, Ryan C. Briggs, Hristos Doucouliagos, Marco Mendoza Aviña, and T.D. Stanley. 2022. “Quantitative Political Science Research Is Greatly Underpowered.” OSF Preprints. July 5. doi: 10.31219/osf.io/7vy2f.
2. Kane, John V. 2023. “More Than Meets the ITT: A Guide for Investigating Null Results .” APSA Preprints. doi: 10.33774/apsa-2023-h4p0q-v2.

I’ve been long interested in statistical power (see Rainey 2014¹ and Rainey 2015²), and these new papers have me thinking even more about the importance of power.

In this post, I argue that statistical power isn’t something ancillary. Power is primary. I also argue that power isn’t something you–the researcher–build to satisfy an especially cranky Reviewer 2, it’s something you do for yourself, to make sure that your study succeeds.

The Hypothesis Testing Framework

In the hypothesis testing framework, you consider two hypotheses: the null hypothesis and the alternative hypothesis.

The hypothesis test is all about arguing against the null hypothesis (leaving the alternative as the only remaining possibility). You will (try to) show that your data would be “unusual” if the null hypothesis were correct.³

If the data would NOT be unusual under the null hypothesis, then you do not reject the null hypothesis.

Intepreting a Failure to Reject

A failure to reject means that the data “would not be unusual under the null hypothesis.” This does not imply that you should conclude the data are only consistent with the null. Indeed, there is a sharp asymmetry in hypothesis testing. I describe this in my 2014 AJPS:

Political scientists commonly interpret a lack of statistical significance (i.e., a failure to reject the null) as evidence for a negligible effect (Gill 1999), but this approach acts as a broken compass… If the sample size is too small, the researcher often concludes that the effect is negligible even though the data are also consistent with large, meaningful effects. This occurs because the small sample leads to a large confidence interval, which is likely to contain both “no effect” and large effects.

Gill (1999)⁴ describes this more forcefully:

We teach graduate students to be very careful when describing the occurrence of not rejecting the null hypothesis. This is because failing to reject the null hypothesis does not rule out an infinite number of other competing research hypotheses. Null hypothesis significance testing is asymmetric: if the test statistic is sufficiently atypical given the null hypothesis then the null hypothesis is rejected, but if the test statistic is insufficiently atypical given the null hypothesis then the null hypothesis is not accepted. This is a double standard: H1 is held innocent until proven guilty and Ho is held guilty until proven innocent (Rozeboom 1960)…

There are two problems that develop as a result of asymmetry. The first is a misinterpretation of the asymmetry to assert that finding a non-statistically significant difference or effect is evidence that it is equal to zero or nearly zero. Regarding the impact of this acceptance error Schmidt (1996: 126) asserts that this: “belief held by many researchers is the most devastating of all to the research enterprise.” This acceptance of the null hypothesis is damaging because it inhibits the exploration of competing research hypotheses. The second problem pertains to the correct interpretation of failing to reject the null hypotheses. Failing to reject the null hypothesis essentially provides almost no information about the state of the world. It simply means that given the evidence at hand one cannot make an assertion about some relationship: all you can conclude is that you can’t conclude that the null was false (Cohen 1962).

There are many incorrect, but somewhat innocent interpretations of p-values. Interpreting a lack of statistical significance as evidence for the null is incorrect and wildly misleading in many cases.

The Implication of a Non-Conclusion

If you cannot draw a conclusion then, what exactly has happened? Obtaining will not be an “error” because you won’t make a strong claim that the research hypothesis is wrong. Instead, you will simply admit that you failed to uncover evidence against the null. Failing to uncover evidence isn’t an error.

Indeed, Jones and Tukey (2000)⁵ write:

A conclusion is in error only when it is “a reversal,” when it asserts one direction while the (unknown) truth is the other direction. Asserting that the direction is not yet established may constitute a wasted opportunity, but it is not an error.

This is worth emphasizing in a different way. Tests are not magical tools that tell you which hypothesis is correct. Instead, tests summarize the evidence against the null. There are two critical pieces to “evidence against the null”: (1) the amount of evidence and (2) whether the evidence is against the null. If you buy your own argument that the null is false (surely you do!), then (2) is taken care of. Only the amount of evidence remains, and you–the researcher–choose the amount of evidence to supply.

The Implication for Power Calculations

This perspective helps motivate power calculations. By their design, tests control the error rate in certain situations (when then null is correct). You do not need to worry about Type I errors. First, the test controls the error rate under the null. Second, you are pretty sure the null is wrong (see your theory section).

If you aren’t worried about Type I errors, what are you worried about? They only thing left to worry about is wasting your time and money. Statistical power is the chance not of wasting your time and money.

Power isn’t a secondary quantity that you compute for thoroughness or in anticipation of a comment from Reviewer 2. Power is something that you build for yourself.

Statisticians talk a lot about Type I errors because that’s their contribution. It’s your job to bring the power.

And importantly, power is under your control. Kane provides a rich summary of ways to increase the power of your experiment. At a minimum, you have brute force control through sample size.

Power isn’t an ancillary concern, it’s the entire game from the very beginning of the planning stage. It should be at the forefront of the researchers mind from the very beginning. You should want the power as high as possible.⁶

You have to supply a test overwhelming evidence to consistently reject the null. Careful power calculations help you make sure you succeed in this war against the null.

Power isn’t about Type S and M errors (Gelman and Carlin 2014)⁷. Power is about you protecting yourself from a failed study. And that seems like a protection worth pursuing carefully.⁸