Why is the power of studies that only report significant effects not always 100%?

Question

As I was reading the following passage of this blog, which defines R-(replicability-)indices:

To correct for the inflation in power, the R-Index uses the inflation rate. For example, if all studies are significant and average power is 75%, the inflation rate is 25% points. The R-Index subtracts the inflation rate from average power. So, with 100% significant results and average observed power of 75%, the R-Index is 50% (75% – 25% = 50%).

..the following question came to my mind:

If statistical power is (please correct me if I am wrong) the sensitivity of a test to detect an effect when it truly exists (and thus indicates the true positive rate), then how can it be any less than 100% when computed on the basis of studies that only report significant effects?

Surely, for all that those studies know, they (think they) have detected some effects out of an unknown total number of effects, and it is unknown to them what other effects (also) "truly" existed but were in fact not detected.

What specific stats reported in a study (one that says nothing of undetected effects) would one need to compute its power? And how is this computation different when done analytically (post-hoc, e.g. by someone else later in a meta-analysis) as opposed to by the experimenters themselves prior to data collection (i.e. as a power analysis)?

I know I am probably confused about many of the key concepts here, so would be grateful for a clarification

Answer 1

I agree that, retrospectively, the power of significant effects is 1 (and the power of nonsignificant effects is 0). After all, power is the probability of rejecting the null, and if you take all information into account, you know what that result is.

There are a number of people who ignore only the one detail about whether they accepted or rejected, and compute power based on the observed effect size, error variance, etc. They obtain a number alright, but it is not a result that adds any information. The reason they do this is along these lines: "I see that the result is nonsignificant, but if I compute the power, I can tell whether that happened because the effect is small (power is high) or the study is under-powered (if the power is low). The problem is that this retrospective power is always low when the test was nonsignificant, and always high when the test was significant. One can go into gyrations explaining that, but basically it is because you are really estimating 0 and 1 respectively, as you have reasoned in your question.

Here are some references that add meat to this.

• Hoenig, J. M. and Heisey, D. M. (2001). The abuse of power: The pervasive fallacy of power calculations in data analysis. The American Statistician, 55:19–24.

• Zumbo, B. D. and Hubley, A. M. (1998). A note on misconceptions concerning prospective and retrospective power. The Statistician, 47(2):385–388.

• Lenth, R. V. (2007). Post Hoc Power: Tables and Commentary. Unpublished manuscript available at https://stat.uiowa.edu/sites/stat.uiowa.edu/files/techrep/tr378.pdf. This article was rejected by a psychology journal because the editor said the results were already well established (which is true, e.g., it cites the above two papers; but the use of post-hoc power remains pervasive and that's why I wrote the paper and why a psychology journal should expose that fallacy).

Answer 2

If statistical power is the sensitivity of a test to detect an effect when it truly exists...

Your misunderstanding arises from whether the effect is at the sample level or the population level.

Suppose we want to test whether playing chess is associated to better memory scores. You get a sample of 30 chess players and 30 non-chess players then administrate a memory test on them. In the sample, the chess players perform significantly better.

Does this imply that, for the population, that chess players significantly out score non-chess players in memory tests? Not necessarily. It may have been that the difference was caused by:

• Systematic error. For example, if the memory test was about remembering chess positions, then the experiment is biased. We don't know much more about population (general) memory from this. Though we do know a little more about population (chess) memory
• Random error. For example, the chess players might have been lucky on the test that day