4
$\begingroup$

I'm wondering whether or not I should adjust the significance level of paired t-tests due to multiple tests (to avoid the possibility of Type 1 error), although the tests are independent. Here's what I'm trying to test:

4 groups of participants each underwent a different mood manipulation procedure. To test the effect of the mood manipulation, I'm using a word recall test, where the amount of correctly recalled positive words is compared to the amount of correctly recalled negative words.

In other words, I'm conducting 4 paired t-tests (number of positive vs. number of negative words), one for each group. Should I correct for multiple tests - and if yes - which correction method would you recommend in this situation?

$\endgroup$
1
  • $\begingroup$ Correction: shouldn't the Bonferroni adjustment be .05/4 ? (p level should be .05 not .5) $\endgroup$ Commented Dec 3, 2016 at 22:09

1 Answer 1

2
$\begingroup$

The first question, of whether you should correct for multiple comparisons, is a tricky one. I think you could go either way. If this is an exploratory procedure, then I would argue that no correction is needed. Also, you are only doing 4 tests, so its not like you are going all out and doing 100 analyses. The nature of the different groups might be important. If these are very different groups, then you could defend that your analysis explores the relationship in different groups (e.g., sex, different races, age groups). I think you can go either way. Another question is does it even matter? If you have significance (or no significance) by adjusting for Type 1 error and without adjusting, then this is a moot point.

As for adjusting, I would likely just do the Bonferroni adjustment. Divide the p value by the number of analyses. In your case, it would likely be .5/4 = .125. If you are doing a one-tailed test (you have an idea of which direction the change should go, which is likely), make sure you are using the one-tailed value. If you are given a two-tailed p value, you can simply divide it in half, I think.

$\endgroup$
3
  • $\begingroup$ Thank you for your answer Behacad! Indeed, the reason I'm asking this is that the statistical significance of the analyses is not that clear. There was a statistically significant difference only in one group, where t(29) = 2.16, p (2-tailed) = .039. Bonferroni correction would make this result non-significant, even if I used one-tailed p (.020). $\endgroup$
    – Jonna
    Commented Apr 24, 2013 at 18:04
  • $\begingroup$ "a one-tailed test (you have an idea of which direction the change should go, which is likely)" --- be careful with this. You should only be using a one-tailed test if an effect in the other direction would be so bizarre/impossible that any difference in that direction, even an enormous one, would still lead you to retain the null. I find there are very few situations where this is truly the case. What you're suggesting here sounds more like using one-tailed tests as a way to relax the significance threshold. $\endgroup$ Commented Dec 3, 2016 at 22:27
  • $\begingroup$ FWIW, it's okay to move your significance threshold (.05 is just a number, after all), but be clear about what you're doing. If you use "one-tailed" tests in situations where you would theoretically interpret a meaningful effect in either direction, then you're just working with an alpha of .1 instead of .05 (i.e. increasing the type 1 error rate). $\endgroup$ Commented Dec 3, 2016 at 22:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.