7
$\begingroup$

In a 2013 review article in Nature Neuroscience, Button et al. Power failure: why small sample size undermines the reliability of neuroscience, it was stated that:

the average statistical power of studies in the neurosciences is very low

They searched for meta-analyses, calculated the post-hoc power in each one of them, and combined the results by taking the median post-hoc power. The median was 20%. I just don't get it. Post-hoc power is always inherently associated with the achieved p-values. Isn't it the same as to write that median p-value was something like ~0.3 which corresponds to the post-hoc power of 20%?

So basically how does this result undermine the quality of research in neurosciences? It seems that they have been publishing studies with a lot of non-significant p-values.

This review is a Nature Neuroscience study with very famous authors, so I think my interpretation is more likely flawed.

EDIT: I would see some point if they included only studies with nominal significance. In that case the median power would tell the median replication probability of significant findings.

|cite|improve this question
$\endgroup$
  • 1
    $\begingroup$ Post hoc power is pointless and misleading. See Hoenig & Heisey (2001, The American Statistician). "It is a Nature study with very famous authors, so I think my interpretation is more likely flawed." - this is a false conclusion. Nature, Science and similar journals select for surprising findings, not for valid statistics. The reviewers here unfortunately typically don't know more statistics than those in lower-ranked journals. $\endgroup$ – S. Kolassa - Reinstate Monica Apr 30 '17 at 18:43
  • 1
    $\begingroup$ I agree that post hoc power analysis of a single study is pointless and misleading. However, I do think that it is meaningful to make a statement about the typical power level of an area of research, as this is not tied to a specific study's finding at an interpretative level. The assumptions of doing so, however, may be questionable if the meta-analysis had heterogeneous results, as is often the case. $\endgroup$ – dbwilson Apr 30 '17 at 18:58
  • 1
    $\begingroup$ @StephanKolassa I agree that there can be junk published in Nature or Science, but I find your comment somewhat unhelpful unless you are familiar with this particular study. In fact IMHO it's a good paper, and they are not doing post hoc power, so OP's understanding is indeed flawed. The answer by dbwilson seems to explain this correctly (+1). $\endgroup$ – amoeba says Reinstate Monica May 2 '17 at 9:53
  • $\begingroup$ It seems that low values of p are being misinterpreted as non-significant . Therefore it is difficult to answer the question. Further confusion is caused by introducing the idea of median in this perspective. The question may be bettered by defining certain concepts. $\endgroup$ – Subhash C. Davar May 3 '17 at 9:55
6
$\begingroup$

I'm not familiar with this particular study but am familiar with estimating the power of an area of research using a meta-analysis. Your statement that "post hoc power is always inherently associated to the achieved p-values" suggests to me that you are assuming that the post hoc power for each individual study contributing to a single meta-analysis is based on the assumption that the observed effect equals the true population effect. Only with that assumption will the post hoc power be related to the p-value (I personally find this form of post hoc power analysis to be pointless, but that is a bit off topic).

What I believe the authors of this paper are doing, given that it is what is typically done, is to assume that the meta-analytic mean is the true population effect and to estimate power for the studies contributing to that mean using that value, not each study's observed effect. Thus, the mean power within a meta-analysis is a function of the overall meta-analytic mean effect and the sample sizes (or standard errors) of the individual studies. They did this for each meta-analysis and computed the median power across them.

|cite|improve this answer
$\endgroup$
  • 1
    $\begingroup$ Out of curiosity I skimmed the paper, and they use G*Power, whose authors explicitly disavow "retrospective power" analyses. (At least in the Faul et al. 2007 paper linked from their site; I have not used the software.) $\endgroup$ – GeoMatt22 Apr 30 '17 at 17:59
  • $\begingroup$ +1. I know the Button et al. 2013 paper, used it in teaching, and I believe this is the correct answer. Cc to @GeoMatt22. If you have one study with n=10 and another with n=1000, and the large study shows that the effect is very small, then it's completely fine to do a "meta-analysis", conclude that the true effect is likely small (based on the large study), and further conclude that the small study must have had very low power - even if it did report a significant p-value and a large effect size. I don't know why anybody would disavow that. $\endgroup$ – amoeba says Reinstate Monica May 2 '17 at 9:51
  • $\begingroup$ @amoeba I am not familiar with this area, and I hope my comment did not come across wrong: It was intended to support the answer's 1st paragraph, i.e. the software the authors used supports post-hoc analysis, but not retrospective analysis. And what OP is calling "post hoc" may be what is more typically called "retrospective". (But again, I am not familiar with the field.) $\endgroup$ – GeoMatt22 May 2 '17 at 14:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.