2
$\begingroup$

I'm conducting research on the treatment effect of tinnitus. Δ represents the pre-treatment to post-treatment change. ΔTHI yielded a p-value of 0.03, while the other p-values are above 0.5.

Now, should I apply the Bonferroni correction to assess whether ΔTHI between the two groups (treatment vs. sham) is significantly different? (This would make the significance threshold p-value 0.05/4 = 0.0125.) Alternatively, can I use a significance threshold of 0.05 without correction? enter image description here

$\endgroup$
2
  • $\begingroup$ This would be a classical case where you would apply some sort of correction, say Bonferroni-Holm, so yes. $\endgroup$ Commented Oct 25, 2023 at 11:51
  • $\begingroup$ Was there a 'primary' outcome specified in your protocol or analysis plan (developed before the data was collected?) $\endgroup$ Commented Oct 25, 2023 at 12:48

1 Answer 1

4
$\begingroup$

The term "significance" is always relative to the significance level, so in order to say whether something is significant, you need to specify that level anyway. The full story here is that "$\Delta$THI is at 5% level significantly different between the groups, however it is no longer significant (at that level) after adjusting for multiple testing". In my view you can say all this, and actually you should say all this (the $p=0.03$ may still indicate something of potential relevance but new data would be required to investigate whether what you observed there has any stability in case such a result would be of interest).

From the fact that a clearly visible difference in the first plot only yields $p=0.03$ I suspect that your sample size is pretty small. Always keep in mind that the issue is not settled by one study alone, regardless of whether you find a significant result or not, and this is even more important to acknowledge if you have a small sample size. Also keep in mind that if indeed nothing at all would be going on that differentiates the treatment from non-treatment, the chance to find (at least) one p-value $<0.05$ in four tests is up to 20%, which is pretty high, so it's quite realistic to see such a result even if in fact nothing is going on. That's the reason multiple testing adjustment should be applied; still looking at the data something may well be going on with $\Delta$THI.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.