# Why is the power of studies that only report significant effects not always 100%?

As I was reading the following passage of this blog, which defines R-(replicability-)indices:

To correct for the inflation in power, the R-Index uses the inflation rate. For example, if all studies are significant and average power is 75%, the inflation rate is 25% points. The R-Index subtracts the inflation rate from average power. So, with 100% significant results and average observed power of 75%, the R-Index is 50% (75% – 25% = 50%).

..the following question came to my mind:

If statistical power is (please correct me if I am wrong) the sensitivity of a test to detect an effect when it truly exists (and thus indicates the true positive rate), then how can it be any less than 100% when computed on the basis of studies that only report significant effects?

Surely, for all that those studies know, they (think they) have detected some effects out of an unknown total number of effects, and it is unknown to them what other effects (also) "truly" existed but were in fact not detected.

What specific stats reported in a study (one that says nothing of undetected effects) would one need to compute its power? And how is this computation different when done analytically (post-hoc, e.g. by someone else later in a meta-analysis) as opposed to by the experimenters themselves prior to data collection (i.e. as a power analysis)?

I know I am probably confused about many of the key concepts here, so would be grateful for a clarification

• Power really only makes sense prospectively. You can hypothesize a target effect size and an estimate of error variance, and use that information to calculate the power as the probability that that your study will detect it. But after the study is completed, you have already detected some effects and failed to detect others. Power doesn't make any sense then. And I think that's the basis for the confusion that you report. Commented Feb 22, 2017 at 19:14
• I do not think that in meta-analysis we usually consider the power of the trials we meta-analyse. Can you clarify that part? Commented Feb 23, 2017 at 14:18
• I can't, since I am not the author of the text I cited. They must have understood power in a different sense to that which you describe @rvl, since they were able to compute it aposteriori. Commented Feb 23, 2017 at 15:30
• There are definitely people - especially in the social sciences - who think they are computing power post-hoc. But they are operating under mistaken premises, and in fact what they are calculating is just a transformation of the P value. I think you have it RIGHT, and I'd hate to see you be lured astray. I'll try to put together an answer with more details and references. Commented Feb 23, 2017 at 21:06
• The R-index here is assessing inflation. If all are significant, but average power of the tests is less than 100 %, this suggests some form of bias in the literature Commented Feb 23, 2017 at 22:33

I agree that, retrospectively, the power of significant effects is 1 (and the power of nonsignificant effects is 0). After all, power is the probability of rejecting the null, and if you take all information into account, you know what that result is.

There are a number of people who ignore only the one detail about whether they accepted or rejected, and compute power based on the observed effect size, error variance, etc. They obtain a number alright, but it is not a result that adds any information. The reason they do this is along these lines: "I see that the result is nonsignificant, but if I compute the power, I can tell whether that happened because the effect is small (power is high) or the study is under-powered (if the power is low). The problem is that this retrospective power is always low when the test was nonsignificant, and always high when the test was significant. One can go into gyrations explaining that, but basically it is because you are really estimating 0 and 1 respectively, as you have reasoned in your question.

Here are some references that add meat to this.

• Hoenig, J. M. and Heisey, D. M. (2001). The abuse of power: The pervasive fallacy of power calculations in data analysis. The American Statistician, 55:19–24.

• Zumbo, B. D. and Hubley, A. M. (1998). A note on misconceptions concerning prospective and retrospective power. The Statistician, 47(2):385–388.

• Lenth, R. V. (2007). Post Hoc Power: Tables and Commentary. Unpublished manuscript available at https://stat.uiowa.edu/sites/stat.uiowa.edu/files/techrep/tr378.pdf. This article was rejected by a psychology journal because the editor said the results were already well established (which is true, e.g., it cites the above two papers; but the use of post-hoc power remains pervasive and that's why I wrote the paper and why a psychology journal should expose that fallacy).

• I am a psychologist and I couldn't agree with you more. The following article in a psych journal says that confidence intervals should be used instead of post-hoc power. apa.org/science/leadership/bsa/statistical/… Unfortunately, that recommendation and other good ones in the article have been largely ignored. Commented Feb 23, 2017 at 23:06
• Great answer @rvl, thank you. And thanks David Lane for your comment too. Commented Feb 24, 2017 at 15:32

If statistical power is the sensitivity of a test to detect an effect when it truly exists...

Your misunderstanding arises from whether the effect is at the sample level or the population level.

Suppose we want to test whether playing chess is associated to better memory scores. You get a sample of 30 chess players and 30 non-chess players then administrate a memory test on them. In the sample, the chess players perform significantly better.

Does this imply that, for the population, that chess players significantly out score non-chess players in memory tests? Not necessarily. It may have been that the difference was caused by:

• Systematic error. For example, if the memory test was about remembering chess positions, then the experiment is biased. We don't know much more about population (general) memory from this. Though we do know a little more about population (chess) memory
• Random error. For example, the chess players might have been lucky on the test that day
• Thanks, that's a useful comment, though not really sure it addresses my specific questions. Commented Feb 22, 2017 at 15:37