# Underpowered studies and minimum effect size

We know that that underpowered studies lead to overestimated effect sizes (conditioned on statistical significance). See here, here, and here for discussions on this topic. Now, let's suppose that the true effect size $e$ is much smaller than the minimum effect size $E$ that we use in the power calculations. As I understand it, this is not necessarily bad, if we're only interested in detecting effect of size $E$ or larger. What this means is that my study is powered appropriately to detect $E$, but underpowered to detect $e$. So my question is: is the effect estimate from this study likely to be overestimated (because we're underpowered w.r.t $e$), or is it fine (because we're well powered w.r.t $E$, which is the effect size we care about)? If the answer is the former, then why it is acceptable to specify a minimum effect size? Wouldn't we always want to specify (our best estimate of) the true effect size?

It depends on the context. For one, if you have good reasons to think the true effect size is much smaller than the smallest effect size of interest (SESOI), and if that means that the SESOI is implausible, you should question if the study is worth conducting in the first place.

On the other hand, assuming the study is worth it (e.g. the SESOI is plausible, but much smaller effect sizes are plausible too), overestimating the true effect size is a risk (that you can assess), however there are other risks to consider too.

If you specify a smaller effect size when carrying out power calculations, it typically requires a larger sample size. If you carry out a study that will cost time or money, or that could hurt living beings, you have to carefully consider if the risks of finding an exaggerated effect size justify increasing the sample size. Otherwise you risk to waste resources or to hurt others unnecessarily. Anyway, you'll typically have limited funds for your study, so there are some limits to consider at some point.

To assess the various risks related to having an underpowered study (relative to what you think the true effect size might be), it may be useful to plot a power curve, as well as a plot of plausible effect sizes vs. the associated exaggeration ratios. It will allow you to better quantify the risks, and weigh them against the other costs or risks associated to increasing the sample size.

If you're in a (neo)Fisherian framework, it may be also useful to plot the effect sizes vs. the median p-values. For an example and details, see section 2.5 from A Reckless Guide to P-values and its figure 5.

Apart from that, there are hybrid Bayesian-frequentist power calculations that allow you to take into account the uncertainty around the estimation of the effect size, taking advantage of prior information you may have and formalizing the belief you have about the effect size.

If you have prior information, besides the hybrid approach mentioned above, you could also consider going full Bayesian, but there are people on this website much more capable than me to give you good answers about that.

Finally, as you're talking about estimating the true effect size, it may be relevant to determine the sample size based on precision of confidence intervals.

Wouldn't we always want to specify (our best estimate of) the true effect size?

YES! Always specify the observed effect and if your best estimate differs from that then you must say how and why. And then discuss what effect sizes would matter and why, noting that that discussion is about issues that are scientific rather than statistical.

Like in so many situations, the all-or-none framework of Neyman–Pearsonian hypothesis tests is unhelpful to clear discussion of the issue raised in the question. Clearly it is possible to have a false negative outcome where the 'real' effect size is too small for any reasonable power to detect. Even if that effect size is too small to be of interest you would usually be better served by estimating it rather than just saying 'not significant'.

Estimation serves as a better framework for thinking about the issue than error rates do. If the true effect size is too small for the experiment to have much power then it is small relative the variability in the measurements. That might be an interesting thing itself. If it is a result of measurement noise then the measuring system might be improved. If it is due to biological variability then it serves to improve the design of future experimentation. Both of those things is scientifically more useful than a 'not significant' nothing to see here, move along type of end point.

I have switched from the term Null hypothesis to the term Main hypothesis. This makes it clearer that you can pick any of the two alternatives as the Main hypothesis which you protect with the choice of $$\alpha$$. Then, you add you choice of power to protect you from the other error risk.

If you do this correctly, you are properly powered to detect $$E$$, and you have decided beforehand that you accept the Type I an II risks. If the true effect size is $$e < E$$, but you end up with an incorrect decision on truth or falsehood, that’s a risk you were willing to take.

Where it goes wrong is if you use the data from the test to calculate a point estimate of the magnitude of the effect. The test has a very non-ambitious goal: it tells you ONLY WHETHER the true effect size $$\geq E$$, accepting incorrect decision at the rates of the Type I an II errors over the course of your entire career. It does NOT tell you by HOW MUCH the effect exceeds $$E$$.

If you use the data from the test to calculate the confidence interval on the point estimate, you take the risk again: there is the inherent risk that you take the incorrect decision. And then you take that same risk again when you calculate the interval. But, this time, that risk is strongly “covariant” to the first risk you take.

This risk-over-risk taking induces the exaggeration error. So, Plan beforehand to get new data for your point estimate (you need a budget for this!). That also mitigates the risk in your hypothesis test.

• I think that switching terminology like that is unhelpful, particularly given that the "main" hypothesis is usually somewhere in the space of alternatives, and can be a non-statistical hypothesis relating to the real world. Commented Jul 15 at 21:49
• In fact, I think it must be related to the real world. A case example is when we perform a study to show that a change in our process improves our production yield. The opportunity cost of missing out on an effective change is vastly higher than the operational costs of the follow-up trial production runs when we continue to pursue a fruitless path. Then, We need to maximize the number of effective treatments we detect, not minimize the number of ineffective treatments we fail to reject. The term Null hypothesis has a strong connotation of Nil hypothesis and I am tired of explaining this.
– W_vH
Commented Jul 16 at 6:24
• In addition, in my language, the word for zero is “nul”. This makes it even more difficult that Null doesn’t need to be Nil. Simply changing terms does a lot more to make people understand this than hundreds of repetitions “Null does not need to be Nil”.
– W_vH
Commented Jul 16 at 6:26
• I think that you have not understood the way that hypothesis testing works, but I agree with your idea that it would be better for people to be more clear that the null hypothesis does not have to be a hypothesis of zero effect. See here: stats.stackexchange.com/questions/447510/… Commented Jul 16 at 20:30
• Well, that’s a bit of a bold statement, and perhaps a bit out of line? Neyman and Pearson had the not-so-ambitious goal “that we shall not be too often wrong”. I am achieving this with my interpretation. Or perhaps you may refer to the real-world comment. Any hypothesis that has no real-world implications is of no interest to me. Hence, my comment.
– W_vH
Commented Jul 16 at 20:54