25
$\begingroup$

This question has been has asked before here and here but I don't think the answers address the question directly.

Do underpowered studies have increased likelihood of false positives? Some news articles make this assertion. For example:

Low statistical power is bad news. Underpowered studies are more likely to miss genuine effects, and as a group they're more likely to include a higher proportion of false positives -- that is, effects that reach statistical significance even though they are not real.

As I understand it, the power of a test can be increased by:

  • increasing the sample size
  • having a larger effect size
  • increasing the significance level

Assuming we don't want to change the significance level, I believe the quote above refers to changing the sample size. However, I don't see how decreasing the sample should increase the number of false positives. To put it simply, reducing the power of a study increases the chances of false negatives, which responds to the question:

$$P(\text{failure to reject }H_{0}|H_{0}\text{ is false})$$

On the contrary, false positives respond to the question:

$$P(\text{reject }H_{0}|H_{0}\text{ is true})$$

Both are different questions because the conditionals are different. Power is (inversely) related to false negatives but not to false positives. Am I missing something?

$\endgroup$
  • 4
    $\begingroup$ It is not the false positive rate that depends on statistical power, but the "false discovery rate": $P(H_0 \text{is true}| \text{reject} H_0)$ $\endgroup$ – Jake Westfall Oct 10 '15 at 21:17
  • 2
    $\begingroup$ Yes, that seems to be the correct interpretation of the statement in the Wired article. $\endgroup$ – Robert Smith Oct 11 '15 at 2:14
32
$\begingroup$

You are correct in that sample size affects power (i.e. 1 - type II error), but not type I error. It's a common misunderstanding that a p-value as such (correctly interpreted) is less reliable or valid when the sample size is small - the very entertaining article by Friston 2012 has a funny take on that [1].

That being said, the issues with underpowered studies are real, and the quote is largely correct I would say, only a bit imprecise in its wording.

The basic problem with underpowered studies is that, although the rate of false positives (type I error) in hypothesis tests is fixed, the rate of true positives (power) goes down. Hence, a positive (= significant) result is less likely to be a true positive in an underpowered study. This idea is expressed in the false discovery rate [2], see also [3]. This seems what the quote refers to.

An additional issue often named regarding underpowered studies is that they lead to overestimated effect sizes. The reasons is that a) with lower power, your estimates of the true effects will become more variable (stochastic) around their true value, and b) only the strongest of those effects will pass the significance filter when the power is low. One should add though that this is a reporting problem that could easily be fixed by discussing and reporting all and not only significant effects.

Finally, an important practical issue with underpowered studies is that low power increases statistical issues (e.g. bias of estimators) as well as the temptation for playing around with variables and similar p-hacking tactics. Using these "researcher degrees of freedom" is most effective when the power is low, and THIS can increase type I error after all, see, e.g., [4].

For all these reasons, I would therefore be indeed skeptical about an underpowered study.

[1] Friston, K. (2012) Ten ironic rules for non-statistical reviewers. NeuroImage, 61, 1300-1310.

[2] https://en.wikipedia.org/wiki/False_discovery_rate

[3] Button, K. S.; Ioannidis, J. P. A.; Mokrysz, C.; Nosek, B. A.; Flint, J.; Robinson, E. S. J. & Munafo, M. R. (2013) Power failure: why small sample size undermines the reliability of neuroscience. Nat. Rev. Neurosci., 14, 365-376

[4] Simmons, J. P.; Nelson, L. D. & Simonsohn, U. (2011) False-Positive Psychology: Undisclosed Flexibility in Data Collection and Analysis Allows Presenting Anything as Significant. Psychol Sci., 22, 1359-1366.

$\endgroup$
  • $\begingroup$ Thank you. Excellent references. For completeness, [1] can be found here and [3] is available here. When you talk about false discovery rate, are you sure that is the right concept? Based on [3], maybe you meant the positive predictive value (PPV) in which underpowered studies have lower PPV (that is, true positives are not as frequent as they should be in a high powered study) It looks like false discovery rate is the complement of PPV. $\endgroup$ – Robert Smith Oct 11 '15 at 1:10
  • $\begingroup$ The way I understand it, these concepts are identical, PPV = 1-FDR. I prefer the use of FDR because I find the word intuitively better understandable. $\endgroup$ – Florian Hartig Oct 11 '15 at 7:51
  • $\begingroup$ See also here en.wikipedia.org/wiki/Positive_and_negative_predictive_values $\endgroup$ – Florian Hartig Oct 11 '15 at 7:53
  • 2
    $\begingroup$ Tal Yarkoni points out all the things wrong about the Friston article here. $\endgroup$ – jona Oct 11 '15 at 14:27
  • 1
    $\begingroup$ @jona - I think Tal Yarkoni raises some good points in his blog post. I guess the 1-sentence summary would be "low power is a problem", which is exactly what I say above. I still find Friston's caricature of reviewer comments funny, because it does happen that reviewers "find the sample size too low" without a cogent argument that involves having calculated power. $\endgroup$ – Florian Hartig Oct 11 '15 at 15:00
6
$\begingroup$

Depending on how you look at it, low power can increase false positive rates in given scenarios.

Consider the following: a researcher tests a treatment. If the test comes back as insignificant, they abandon it and move onto the next treatment. If the test comes back significant, they publish it. Let's also consider that the researcher will tests some treatments that work and some that don't. If the researcher has high power (of course referring to the case when they are testing a treatment that works), then they are very likely to stop once they test an effective treatment. On the other hand, with low power, they are likely to miss the true treatment effect and move on to other treatments. The more null treatments they test, the more likely they are to make a Type I error (this researcher does not account for multiple comparisons). In the case of low power, they are expected to test many more null treatments, thus greatly increasing the chance that this researcher will make a type I error.

You might say "well, this is just a researcher abusing multiple comparisons!". Well, that may be true, but that is also how a lot of research gets done these days. Because of exactly these reasons, I personally have little faith in published work unless it has a large enough sample size such that the researcher could not afford to repeat the same experiment a large number of times.

$\endgroup$
  • 1
    $\begingroup$ Thank you. Even ignoring the case of multiple comparisons (without proper corrections), I think you're describing another instance of PPV as described here. I can't paste the paragraph but it begins with (For example, suppose that we work in a scientific field in which one in five of the effects we test are expected to be truly non-null) $\endgroup$ – Robert Smith Oct 11 '15 at 1:30
  • 1
    $\begingroup$ Ah yes, that very closely describes what I was referring to. The smallest distinction is that I'm saying "In a given experimental procedure, having individual low power at each test of a true effect increases the odds of using making a type I error in our entire experimental procedure". This is, of course, different than increasing the type I error rate in each statistical test. Also, it is only in the most technical of senses different than PPV. But it's the only way the media statement "low power increases type I errors" makes sense (and I think it makes a lot of sense). $\endgroup$ – Cliff AB Oct 11 '15 at 1:40
4
$\begingroup$

Low power can't effect the Type-1 error rate, but it could effect the proportion of published results that are type-1 errors.

The reason is that low power reduces the chances of a correct rejection of H0 (Type-2 error) but not the chances of a false rejection of H0 (Type-1 error).

Assume for a second that there are two literatures...one conducted with very low power -- near zero -- and the other conducted with adequate power. In both literatures, you can assume that when H0 is false, you will still get false positives some of the time (e.g., 5% for alpha = .05). Assuming researchers are not always correct in their hypotheses, we can assume both literatures should have a similar NUMBER of Type-1 errors, good power or not. This is because the rate of Type-1 errors is not impacted by power, as other have said.

However, in the literature with LOW power, you would also have a lot of Type-2 errors. In other words, the low-power literature should LACK correct rejections of H0, making the Type-1 errors a larger proportion of the literature. In the high-power literature, you should have a mixture of correct and incorrect rejections of H0.

So, does low power increase Type-1 errors? No. However, it does make it harder to find true effects, making Type-1 errors a larger proportion of published findings.

$\endgroup$
  • 1
    $\begingroup$ Thank you. What about the PPV? In the paper referenced by Florian Hartig, there is the claim that given a type I error, the lower the power, the lower the PPV. If the PPV is lower, which means that the number of true claimed discoveries is lower, then the number of false claimed discoveries (false positives) should increase. $\endgroup$ – Robert Smith Oct 11 '15 at 1:19
0
$\begingroup$

In addition to the others answer, a study is usually underpowered when the sample size is small. There are many tests that are only asymptotically valid, and too optimistic or conservative for small n.

Other tests are only valid for small sample sizes if certain conditions are met, but become more robust with a large sample size (e.g. t-test).

In both these cases small sample size and unmet assumption can lead to an increased type I error rate. Both these situations occur often enough that I consider the real answer to your question to be: not in theory but in practice.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.