# What does a post hoc power analysis say about the significance of the study results?

I orginially posted this over at AskMetafilter, and a commenter suggested I ask it here.

I work for a dietary supplement company that also makes skin care products, and some of those skin care products are tested clinically. Now they are talking about repeating some of the clinical tests in another region of the world in which the products will be sold because the marketing department thinks that would be good. The question came up: How do you decide how many subjects to include?

The answer, I learn, is statistical power. You include at least as many subjects as are needed to give you an appropriate power, say 80%, given your chosen significance level alpha and expected effect size, for the type of statistical test you are performing. What if p is small, lower than alpha, but power is also low? Does that invalidate the results?

For example, one of the studies they want to repeat used a two-tailed paired t test to compare before and after treatment means for a measurement. Alpha was 0.05 and population size was 30. The pooled SD for the two data sets was 0.688. After the fact, I calculated a Cohen's d of 0.494. All this gives a power of less than 50%, which means the study was underpowered. At the same time, p was 0.000004.

I can tell the people at work that, when the study is repeated, we are going to need more subjects or we risk missing the effect that we saw in the first trial, but what can we conclude about that first trial? Power was low, but p was much lower than alpha. Are the results no good? Or can we still trust that p? Or, what is also possible, am I completely confused and all this doesn't work the way I think?

Thanks for any help you can provide!

• What was the SD for the effect size? – Ellis Valentiner Jul 27 '14 at 19:38

There are many related posts you may want to look at.

In genearl, I like to think of power like the resolution of a photo. So we might ask, how much resolution do you need to see the picture clearly? or, How much data do you need to see whether there are differences?

Looking at an image with coarse resolution (low power), it is hard to tell what you are seeing. But it is easier if the effect size is large or the image displays a very clear pattern. In conducting power analyses, I think it is usually best to fix the effect size at some meaningful value – the minimum effect size worth detecting.

The question becomes how many observations do I need to reliably detect this difference. This is like using the camera on my phone vs. an expensive DSLR camera. The more expensive camera might take higher resolution pictures and give me more fine detail – but chances are I can tell what I am looking at with the camera on my phone. In statistics, you are calculating the number of observations to achieve the specified power and alpha given that exact effect size. But if the actual effect size is greater, then you actually have more power.

I would suggest re-running your sample size calculations, but using the minimum effect size worth detecting – NOT the effect size you have already observed. This is because you should expect variation in the effect size and you may want to be able to detect an effect size smaller than what you have already observed.

Lastly, you did not include the standard deviation in your original post. If the standard deviation is sufficiently small, then your initial study may have been sufficiently powered.

In R:

> power.t.test(n =  15, delta = 0.49, sd = 0.46, sig.level = 0.05, type="two.sample")

#      Two-sample t test power calculation
#
#              n = 15
#          delta = 0.49
#             sd = 0.66
#      sig.level = 0.05
#          power = 0.8039799
#    alternative = two.sided
#
# NOTE: n is number in *each* group


$p$-value and power are very different in that the power is in fact a probability whereas the $p$-value is only some number from $[0,1]$. This number represents both the effect size and the random outcome of the data. That means that even with a huge effect, you can still have a $p$-value close to 1: Bad luck. So in the end there is no insightful relationship between a single $p$-value and the power. In other words, $p$-values are hard to reproduce. Test results in turn are easier to reproduce. So only consider if $p\leq\alpha$, nothing else.

For your power calculation, I would suggest also to look at the "guesses" of the previous trial's power calculation. I would tend to use the trial's data only in order to update the assumed standard deviation. The absolute effect size should remain "external", i.e. from some reasoning about a reasonable relevant effect size.

• Just to add a bit of concreteness to this answer and the one from @user12202013, the effect size should be something really meaningful to the people wanting the study. Talk to both the skin-care people and the marketing people; get them to tell you how big a difference is really meaningful -- to the customer, to serve the marketing needs, etc. And how big can a difference be and still be not worthwhile? BTW, your previous study was not underpowered: you found significance, so obviously you had enough power for that to happen. – Russ Lenth Jul 28 '14 at 0:48
• +1 for stressing the practical meaning of effect size. However, saying the power was enough since we rejected $H_0$ is a bit too bayesian. A frequentist would say, we just don't know, its random by assumption. – Horst Grünbusch Jul 28 '14 at 7:37
• Isn't the $p$-value a true probability? Taken, one that does depend on the sample. It's the probability to obtain a sample at least as unlikely as the considered one, given the null is true. – ziggystar Jul 28 '14 at 7:56
• Hmmmm, I haven't been called too Bayesian very many times :). I think I stated it awkwardly, but the point is that power is the probability of rejecting $H_0$, and since that was done, then retrospectively the power is 1. – Russ Lenth Jul 28 '14 at 12:13