Do underpowered studies have increased likelihood of false positives?

Question

This question has been has asked before here and here but I don't think the answers address the question directly.

Do underpowered studies have increased likelihood of false positives? Some news articles make this assertion. For example:

Low statistical power is bad news. Underpowered studies are more likely to miss genuine effects, and as a group they're more likely to include a higher proportion of false positives -- that is, effects that reach statistical significance even though they are not real.

As I understand it, the power of a test can be increased by:

• increasing the sample size
• having a larger effect size
• increasing the significance level

Assuming we don't want to change the significance level, I believe the quote above refers to changing the sample size. However, I don't see how decreasing the sample should increase the number of false positives. To put it simply, reducing the power of a study increases the chances of false negatives, which responds to the question:

$$P(\text{failure to reject }H_{0}|H_{0}\text{ is false})$$

On the contrary, false positives respond to the question:

$$P(\text{reject }H_{0}|H_{0}\text{ is true})$$

Both are different questions because the conditionals are different. Power is (inversely) related to false negatives but not to false positives. Am I missing something?

Answer 1

You are correct in that sample size affects power (i.e. 1 - type II error), but not type I error. It's a common misunderstanding that a p-value as such (correctly interpreted) is less reliable or valid when the sample size is small - the very entertaining article by Friston 2012 has a funny take on that [1].

That being said, the issues with underpowered studies are real, and the quote is largely correct I would say, only a bit imprecise in its wording.

The basic problem with underpowered studies in the standard NHST framework is that, although the rate of false positives (type I error) is fixed, the rate of true positives (power) goes down, and thus the relative proportion of true positives in all positives. Hence, if we assume that people tend to publish only positive (= significant) results, a reported positive result is less likely to be a true positive when the study is underpowered. This idea is expressed in the false discovery rate [2], see also [3]. This seems what the quote refers to.

An additional issue often named regarding underpowered studies is that they lead to overestimated effect sizes. The reasons is that a) with lower power, your estimates of the true effects will become more variable (stochastic) around their true value, and b) only the strongest of those effects will pass the significance filter when the power is low. One should add though that this is a reporting problem that could easily be fixed by discussing and reporting all and not only significant effects.

Finally, an important practical issue with underpowered studies is that low power increases statistical issues (e.g. bias of estimators) as well as the temptation for playing around with variables and similar p-hacking tactics. Using these "researcher degrees of freedom" is most effective when the power is low, and THIS can increase type I error after all, see, e.g., [4].

For all these reasons, I would therefore be indeed skeptical about an underpowered study.

[1] Friston, K. (2012) Ten ironic rules for non-statistical reviewers. NeuroImage, 61, 1300-1310.

[2] https://en.wikipedia.org/wiki/False_discovery_rate

[3] Button, K. S.; Ioannidis, J. P. A.; Mokrysz, C.; Nosek, B. A.; Flint, J.; Robinson, E. S. J. & Munafo, M. R. (2013) Power failure: why small sample size undermines the reliability of neuroscience. Nat. Rev. Neurosci., 14, 365-376

[4] Simmons, J. P.; Nelson, L. D. & Simonsohn, U. (2011) False-Positive Psychology: Undisclosed Flexibility in Data Collection and Analysis Allows Presenting Anything as Significant. Psychol Sci., 22, 1359-1366.

Answer 2

Depending on how you look at it, low power can increase false positive rates in given scenarios.

Consider the following: a researcher tests a treatment. If the test comes back as insignificant, they abandon it and move onto the next treatment. If the test comes back significant, they publish it. Let's also consider that the researcher will tests some treatments that work and some that don't. If the researcher has high power (of course referring to the case when they are testing a treatment that works), then they are very likely to stop once they test an effective treatment. On the other hand, with low power, they are likely to miss the true treatment effect and move on to other treatments. The more null treatments they test, the more likely they are to make a Type I error (this researcher does not account for multiple comparisons). In the case of low power, they are expected to test many more null treatments, thus greatly increasing the chance that this researcher will make a type I error.

You might say "well, this is just a researcher abusing multiple comparisons!". Well, that may be true, but that is also how a lot of research gets done these days. Because of exactly these reasons, I personally have little faith in published work unless it has a large enough sample size such that the researcher could not afford to repeat the same experiment a large number of times.

Answer 3

Low power can't effect the Type-1 error rate, but it could effect the proportion of published results that are type-1 errors.

The reason is that low power reduces the chances of a correct rejection of H0 (Type-2 error) but not the chances of a false rejection of H0 (Type-1 error).

Assume for a second that there are two literatures...one conducted with very low power -- near zero -- and the other conducted with adequate power. In both literatures, you can assume that when H0 is false, you will still get false positives some of the time (e.g., 5% for alpha = .05). Assuming researchers are not always correct in their hypotheses, we can assume both literatures should have a similar NUMBER of Type-1 errors, good power or not. This is because the rate of Type-1 errors is not impacted by power, as other have said.

However, in the literature with LOW power, you would also have a lot of Type-2 errors. In other words, the low-power literature should LACK correct rejections of H0, making the Type-1 errors a larger proportion of the literature. In the high-power literature, you should have a mixture of correct and incorrect rejections of H0.

So, does low power increase Type-1 errors? No. However, it does make it harder to find true effects, making Type-1 errors a larger proportion of published findings.

Answer 4

In addition to the others answer, a study is usually underpowered when the sample size is small. There are many tests that are only asymptotically valid, and too optimistic or conservative for small n.

Other tests are only valid for small sample sizes if certain conditions are met, but become more robust with a large sample size (e.g. t-test).

In both these cases small sample size and unmet assumption can lead to an increased type I error rate. Both these situations occur often enough that I consider the real answer to your question to be: not in theory but in practice.

Answer 5

It's surely very counter-intuitive. There are some great answers in this thread.

We expect that "low statistical power" means the treatment effect has to be truly big to be considered statistically significant. So there must be a drop in FPR (False positive rate). But many are claiming otherwise.

One possible explanation could be that the lower-probability higher-random variations make their way into the positives. Whereas genuine treatment effects get discarded as negatives. In other words, both genuine treatment effect and low-probability random variations start competing to cross the statsig boundary and thereby FPR increases.