Is a priori power analysis essentially useless?

Question

I attended a meeting of the Society for Personality and Social Psychology last week where I saw a talk by Uri Simonsohn with the premise that using an a priori power analysis to determine sample size was essentially useless because its results are so sensitive to assumptions.

Of course, this claim goes against what I was taught in my methods class and against the recommendations of many prominent methodologists (most notably Cohen, 1992), so Uri presented some evidence bearing on his claim. I have attempted to recreate some of this evidence below.

For simplicity, let's imagine a situation where you have two groups of observations and guess that the effect size (as measured by the standardized mean difference) is $.5$. A standard power calculation (done in R using the pwr package below) will tell you will need $128$ observations to obtain 80% power with this design.

require(pwr)

size <- .5
# Note that the output from this function tells you the required observations per group
# rather than the total observations required
pwr.t.test(d = size, 
           sig.level = .05, 
           power = .80, 
           type = "two.sample", 
           alternative = "two.sided")

Usually, however, our guesses about the anticipated size of the effect are (at least in the social sciences, which is my field of study) just that -- very rough guesses. What happens then if our guess about the size of the effect is a little off? A quick power calculation tells you that if the size of the effect is $.4$ instead of $.5$, you need $200$ observations -- $1.56$ times the number that you would need to have adequate power for an effect size of $.5$. Likewise, if the size of the effect is $.6$, you only need $90$ observations, 70% of what you would need to have adequate power to detect an effect size of $.50$. Practically speaking, the range in the estimated observations is quite large -- $90$ to $200$.

One response to this problem is that, instead of making a pure guess as to what the size of the effect might be, you gather evidence about the size of the effect, either through past literature or through pilot testing. Of course, if you're doing pilot testing, you would want your pilot test to be sufficiently small that you're not simply running a version of your study just to determine the sample size needed to run the study (i.e., you'd want the sample size used in the pilot test to be smaller than the sample size of your study).

Uri Simonsohn argued that pilot testing for the purpose of determining the effect size used in your power analysis is useless. Consider the following simulation that I ran in R. This simulation assumes that the population effect size is $.5$. It then conducts $1000$ "pilot tests" of size 40 and tabulates the recommended $N$ from each of the 10000 pilot tests.

set.seed(12415)

reps <- 1000
pop_size <- .5
pilot_n_per_group <- 20
ns <- numeric(length = reps)

for(i in 1:reps)
{
  x <- rep(c(-.5, .5), pilot_n_per_group)
  y <- pop_size * x + rnorm(pilot_n_per_group * 2, sd = 1)
  # Calculate the standardized mean difference
  size <- (mean(y[x == -.5]) - mean(y[x == .5])) / 
          sqrt((sd(y[x == -.5])^2 + sd(y[x ==.5])^2) / 2)

  n <- 2 * pwr.t.test(d = size,
                      sig.level = .05, 
                      power = .80,
                      type = "two.sample", 
                      alternative = "two.sided")$n

  ns[i] <- n
}

Below is a density plot based on this simulation. I have omitted $204$ of the pilot tests that recommended a number of observations above $500$ to make the image more interpretable. Even focusing on the less extreme results of the simulation, there is huge variation in the $Ns$ recommended by the $1000$ pilot tests.

Of course, I'm sure the sensitivity to assumptions problem only gets worse as one's design becomes more complicated. For example, in a design requiring the specification of a random effects structure, the nature of the random effects structure will have dramatic implications for the power of the design.

So, what do you all think of this argument? Is a priori power analysis essentially useless? If it is, then how should researchers plan the size of their studies?

Answer 1

The basic issue here is true and fairly well known in statistics. However, his interpretation / claim is extreme. There are several issues to be discussed:

First, power doesn't change very fast with changes in $N$. (Specifically, it changes as a function of $\sqrt N$, so to halve the standard deviation of your sampling distribution, you need to quadruple your $N$, etc.) However, power is quite sensitive to effect size. Moreover, unless your estimated power is $50\%$, the change in power with a change in effect size isn't symmetrical. If you are trying for $80\%$ power, power will decrease more rapidly with a decrease in Cohen's $d$ than it will increase with an equivalent increase in Cohen's $d$. For example, when starting from $d = .5$ with $N = 128$, if you had 20 fewer observations, power would drop by $\approx 7.9\%$, but if you had 20 more observation, power would increase by $\approx 5.5\%$. On the other hand, if the true effect size were $.1$ lower, then power would be $\approx 16.9\%$ lower, but if it were $.1$ higher, it would only be $\approx 12.6\%$ higher. This asymmetry, and the differing sensitivity, can be seen in the figures below.

If you are working from effects estimated from prior research, say a meta-analysis or a pilot study, the solution to this is to incorporate your uncertainty about the true effect size into your power calculation. Ideally, this would involve integrating over the entire distribution of possible effect sizes. This is probably a bridge too far for most applications, but a quick and dirty strategy is to calculate the power at several possible effect sizes, your estimated Cohen's $d$ plus or minus 1 and 2 standard deviations, and then get a weighted average using the probability densities of those quantiles as the weights.

If you are conducting a study of something that has never been studied before, this doesn't really matter. You know what effect size you care about. In reality, the effect is either that large (or larger), or it is smaller (even possibly 0). Using the effect size you care about in your power analysis will be valid, and will give you an appropriate test of your hypothesis. If the effect size that you care about is the true value, you will have (say) an $80\%$ chance of 'significance'. If, due to sampling error, the realized effect size in your study is smaller (larger) your result will be less (more) significant, or even non-significant. That is the way it is supposed to work.

Second, regarding the broader claim that power analyses (a-priori or otherwise) rely on assumptions, it is not clear what to make of that argument. Of course they do. So does everything else. Not running a power analysis, but just gathering an amount of data based on a number you picked out of a hat, and then analyzing your data, will not improve the situation. Moreover, your resulting analyses will still rely on assumptions, just as all analyses (power or otherwise) always do. If instead you decide that you will continue to collect data and re-analyze them until you get a picture you like or get tired of it, that will be much less valid (and will still entail assumptions that may be invisible to the speaker, but that exist nonetheless). Put simply, there is no way around the fact that assumptions are being made in research and data analysis.

You may find these resources of interest:

• Kraemer, H.C., Mintz, J., Noda, A., Tinklenberg, J., & Yesavage, J.A. (2006). Caution regarding the use of pilot studies to guide power calculations for study proposals, Archives of General Psychiatry, 63, 5, pp. 484-489.

• Uebersax, J.A. (2007). Bayesian Unconditional Power Analysis. http://www.john-uebersax.com/stat/bpower.htm