Take the 2-minute tour ×
Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It's 100% free, no registration required.

I attended a meeting of the Society for Personality and Social Psychology last week where I saw a talk by Uri Simonsohn with the premise that using an a priori power analysis to determine sample size was essentially useless because its results are so sensitive to assumptions.

Of course, this claim goes against what I was taught in my methods class and against the recommendations of many prominent methodologists (most notably Cohen, 1992), so Uri presented some evidence bearing on his claim. I have attempted to recreate some of this evidence below.

For simplicity, let's imagine a situation where you have two groups of observations and guess that the effect size (as measured by the standardized mean difference) is $.5$. A standard power calculation (done in R using the pwr package below) will tell you will need $128$ observations to obtain 80% power with this design.

require(pwr)

size <- .5
# Note that the output from this function tells you the required observations per group
# rather than the total observations required
pwr.t.test(d = size, 
           sig.level = .05, 
           power = .80, 
           type = "two.sample", 
           alternative = "two.sided")

Usually, however, our guesses about the anticipated size of the effect are (at least in the social sciences, which is my field of study) just that -- very rough guesses. What happens then if our guess about the size of the effect is a little off? A quick power calculation tells you that if the size of the effect is $.4$ instead of $.5$, you need $200$ observations -- $1.56$ times the number that you would need to have adequate power for an effect size of $.5$. Likewise, if the size of the effect is $.6$, you only need $90$ observations, 70% of what you would need to have adequate power to detect an effect size of $.50$. Practically speaking, the range in the estimated observations is quite large -- $90$ to $200$.

One response to this problem is that, instead of making a pure guess as to what the size of the effect might be, you gather evidence about the size of the effect, either through past literature or through pilot testing. Of course, if you're doing pilot testing, you would want your pilot test to be sufficiently small that you're not simply running a version of your study just to determine the sample size needed to run the study (i.e., you'd want the sample size used in the pilot test to be smaller than the sample size of your study).

Uri Simonsohn argued that pilot testing for the purpose of determining the effect size used in your power analysis is useless. Consider the following simulation that I ran in R. This simulation assumes that the population effect size is $.5$. It then conducts $1000$ "pilot tests" of size 40 and tabulates the recommended $N$ from each of the 10000 pilot tests.

set.seed(12415)

reps <- 1000
pop_size <- .5
pilot_n_per_group <- 20
ns <- numeric(length = reps)

for(i in 1:reps)
{
  x <- rep(c(-.5, .5), pilot_n_per_group)
  y <- pop_size * x + rnorm(pilot_n_per_group * 2, sd = 1)
  # Calculate the standardized mean difference
  size <- (mean(y[x == -.5]) - mean(y[x == .5])) / 
          sqrt((sd(y[x == -.5])^2 + sd(y[x ==.5])^2) / 2)

  n <- 2 * pwr.t.test(d = size,
                      sig.level = .05, 
                      power = .80,
                      type = "two.sample", 
                      alternative = "two.sided")$n

  ns[i] <- n
}

Below is a density plot based on this simulation. I have omitted $204$ of the pilot tests that recommended a number of observations above $500$ to make the image more interpretable. Even focusing on the less extreme results of the simulation, there is huge variation in the $Ns$ recommended by the $1000$ pilot tests.

enter image description here

Of course, I'm sure the sensitivity to assumptions problem only gets worse as one's design becomes more complicated. For example, in a design requiring the specification of a random effects structure, the nature of the random effects structure will have dramatic implications for the power of the design.

So, what do you all think of this argument? Is a priori power analysis essentially useless? If it is, then how should researchers plan the size of their studies?

share|improve this question
6  
This sounds like a condemnation of mindless power analysis, not of power analysis itself. The bigger question is whether this is an attack on a straw man or if indeed there are many people who conduct their power analyses (or any other analyses) without regard for their sensitivity to assumptions. If the latter is true it is good to enlighten them, but I hope they do not become so discouraged as to abandon all efforts to plan their experiments! –  whuber Feb 24 at 19:05
2  
Reminds me quite a lot of stats.stackexchange.com/q/2492/32036, and not just because of the syntactical similarity in phrasing the title question. Seems a question of how to understand the assumptions. A major point in both is to understand these analyses' sensitivities to bias rather than making sweeping, all-or-nothing judgments that their assumptions are (a) absolutely crucial or (b) utterly negligible. This is key to useful-and-not-harmful inference in general. I'm afraid it's no straw man; people think in absolutes all too often when they don't or can't know or care. –  Nick Stauner Feb 24 at 19:55
3  
I didn't want to add this in the question because I was interested in the recommendations that others made, but Uri Simonsohn's recommendation at the end of the talk was to power your study to detect the smallest effect that you'd care about. –  Patrick S. Forscher Feb 24 at 20:14
5  
@PatrickS.Forscher: So after all is said and done, he does believe in an a priori power analysis. He just thinks the effect size should be chosen wisely: not a guess as to what it might be, but rather the minimal value you'd care about. Sounds pretty much like the textbook description of power analysis: making sure that you have enough data that what you believe to be a practically significant difference will show up as a statistically significant difference. –  Wayne Feb 24 at 22:16
1  
The way Uri framed the talk, I think he believes that a priori power analysis is useless as it's usually done in the social sciences, but perhaps not as it's taught elsewhere. Indeed, I was taught to base my power analysis on a reasonable guess about the size of the effect I'm looking for, not on what effect I would care about in practical terms. –  Patrick S. Forscher Feb 24 at 23:44

1 Answer 1

up vote 12 down vote accepted

The basic issue here is true and fairly well known in statistics. However, his interpretation / claim is extreme. There are several issues to be discussed:

First, power doesn't change very fast with changes in $N$. (Specifically, it changes as a function of $\sqrt N$, so to halve the standard deviation of your sampling distribution, you need to quadruple your $N$, etc.) However, power is quite sensitive to effect size. Moreover, unless your estimated power is $50\%$, the change in power with a change in effect size isn't symmetrical. If you are trying for $80\%$ power, power will decrease more rapidly with a decrease in Cohen's $d$ than it will increase with an equivalent increase in Cohen's $d$. For example, when starting from $d = .5$ with $N = 128$, if you had 20 fewer observations, power would drop by $\approx 7.9\%$, but if you had 20 more observation, power would increase by $\approx 5.5\%$. On the other hand, if the true effect size were $.1$ lower, then power would be $\approx 16.9\%$ lower, but if it were $.1$ higher, it would only be $\approx 12.6\%$ higher. This asymmetry, and the differing sensitivity, can be seen in the figures below.

enter image description here

If you are working from effects estimated from prior research, say a meta-analysis or a pilot study, the solution to this is to incorporate your uncertainty about the true effect size into your power calculation. Ideally, this would involve integrating over the entire distribution of possible effect sizes. This is probably a bridge too far for most applications, but a quick and dirty strategy is to calculate the power at several possible effect sizes, your estimated Cohen's $d$ plus or minus 1 and 2 standard deviations, and then get a weighted average using the probability densities of those quantiles as the weights.

If you are conducting a study of something that has never been studied before, this doesn't really matter. You know what effect size you care about. In reality, the effect is either that large (or larger), or it is smaller (even possibly 0). Using the effect size you care about in your power analysis will be valid, and will give you an appropriate test of your hypothesis. If the effect size that you care about is the true value, you will have (say) an $80\%$ chance of 'significance'. If, due to sampling error, the realized effect size in your study is smaller (larger) your result will be less (more) significant, or even non-significant. That is the way it is supposed to work.

Second, regarding the broader claim that power analyses (a-priori or otherwise) rely on assumptions, it is not clear what to make of that argument. Of course they do. So does everything else. Not running a power analysis, but just gathering an amount of data based on a number you picked out of a hat, and then analyzing your data, will not improve the situation. Moreover, your resulting analyses will still rely on assumptions, just as all analyses (power or otherwise) always do. If instead you decide that you will continue to collect data and re-analyze them until you get a picture you like or get tired of it, that will be much less valid (and will still entail assumptions that may be invisible to the speaker, but that exist nonetheless). Put simply, there is no way around the fact that assumptions are being made in research and data analysis.

You may find these resources of interest:

share|improve this answer
1  
I think Uri Simonsohn's argument wasn't that assumptions per se are bad, but that power analyses in general are so sensitive to assumptions as to make them useless for planning sample sizes. However, your points are excellent, as are the references that you provided (+1). –  Patrick S. Forscher Feb 24 at 20:12
    
Your edits continue to improve this already excellent answer. :) –  Patrick S. Forscher Feb 24 at 23:44

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.