Learning from the flaws in the NHST and p-values

Question

I have been reading a lot about problems with null hypothesis significance testing (NHST) and p-values: the replication crisis, reproducibility issues, p-hacking, issues with low statistical power, misunderstandings of p-values, etc.

However, it is difficult for me to actually turn these concerns into action, especially regarding the flaws in the NHST. I just have the feeling that in biomedical research where both statistical and clinical significance have a role, the view that the issues with NHST are not that big a deal is reasonable.

Let's say I perform a randomized clinical trial in which I aim to investigate the effect of an unnamed drug. I compare it to a placebo. My main interest is some continuous variable, say pain in VAS scale or systolic blood pressure. I measure it after drug/placebo administration.

Let's put p-hacking, data dredging all other issues aside and say that in the placebo group the drug effect is $\mu_1$ and in the other group it is $\mu_2$ when compared to the baseline value. SDs are same in both groups.

1. I perform a test. $H_0$ states that $\mu_1=\mu_2$. I get a p-value of 0.0001. According to the NHST, I reject H0 and conclude that $\mu_1\ne \mu_2$, which is defined by $H_1$. I understand quite well the literature about the flaws in this reasoning: The p-value obtained has nothing to do with $H_1$, the defined $H_1$ is nonsense, it could have been anything, etc. But if I find a mean difference, say, in pain, using the VAS of 0.5cm I can conclude that my result is clinically insignificant since a dozen studies have shown that patient can only notice a change of 1cm in VAS scale. So does this neutralize the issues with NHST?

2. If I do not perform a significance test, what would be most appropriate method to compare these groups? An obvious answer is to use a Bayesian framework, but currently I would prefer other ways. Would bootstrapping or a likelihood ratio test work? And how are these performed in this sort of simple case with a mean comparison? I understand that I should estimate uncertainty much more than to come up with a binary yes/no answer thereby contributing to the p-value fallacy. I just don't understand clearly how to achieve it.

Answer 1

Even if you do get a significant p-value from a test of significance, you are supposed to look at the magnitude of the effect by constructing a confidence interval for it.

Case 1:

When examining the confidence interval, if you notice for example that the interval falls entirely below your predefined threshold for what constitutes a clinically important effect, you will have to conclude that while there is likely an effect in the underlying population, its magnitude is so small from a clinical perspective that it may not have any practical value.

Case 2

If the interval does include your predefined threshold for what constitutes a clinically important effect, you will not be able to rule out the possibility that the effect you are interested in for the underlying population may be large enough to have practical importance.

————-/————-

There is currently a push to replace tests of significance entirely with confidence intervals and, when using confidence intervals, to interpret them as compatibility intervals.

Often, in practice, people will just perform a test of significance, without reporting a confidence interval for the effect they test. When they do report a confidence interval, they might not always have a clear sense of what constitutes a clinically (or practically) relevant effect size. So they might be happy to declare the effect is statistically significant (i.e., it is likely to exist in the underlying population) when its corresponding p-value is significant, without worrying about its clinical (or practical) relevance.

For people who rely exclusively on reporting p-values for tests of significance, there is also the danger that they will invariably conclude a lack of effect when a p-value is not significant when an alternative explanation might be that there is an effect but the study was powered inadequately to detect it. A confidence interval would be easier to interpret in these cases.

In general, the real problem is that people are wired to want definitive findings from one-off studies which are typically under-powered. So when they find inconclusive findings, they stretch them to be more conclusive by using ill-fated terms such as “the result is trending towards statistical significance”, etc.

So I would say you can improve your current practice by always reporting confidence intervals for the effects you test (either in lieu of or as a supplement to tests of significance).

Answer 2

I am going to change your question slightly to "patient can only notice a change of more than 1 cm" (this makes the null a closed set, but a more complicated argument holds for open sets). Your reasoning does not overcome the issue because what you really want to test is $|\mu_0-\mu_1|\le 1{\rm cm}$ and not $\mu_o=\mu_1$, so the standard problems of NHST will come up for the hypothesis $|\mu_0-\mu_1|\le 1{\rm cm}$.

Regardless, an almost sure hypothesis testing resolves these issues in any sufficiently large sample. Take your significance level to be $n^{-p}$ for $p>1$ where $n$ is sample size, then in any sufficiently large sample you will reject $\mu_o=\mu_1$ when it is false and accept $\mu_o=\mu_1$ when it is true with probability one. The same is true for the hypothesis $|\mu_0-\mu_1|\le 1{\rm cm}$. Almost sure hypothesis testing is robust to optional stopping, finitely many multiple comparisons, and publication bias. It can also be used for model selection. I wrote about this in a paper called, "Almost sure hypothesis testing and a resolution of the Jeffreys-Lindley paradox".