2
$\begingroup$

I am comparing two groups of people: Group $A$ and group $B$. For each group, I measured three variables: $v1$, $v2$ and $v3$, all are numeric. My intention is to compare the two groups based on the overall score ($v= v1+v2+v3$), $H_{0}: \mu_{vA} = \mu_{vB}$ and also by the subscore ($H_{0}: \mu_{v1A} = \mu_{v1B}, H_{0}: \mu_{v2A} = \mu_{v2B}, H_{0}: \mu_{v3A} = \mu_{v3B}$). I will be doing a two sample independent T-Test for testing these hypotheses. To me it seemed like these individual comparisons of subscore will not pose a problem of multiple comparisons since I am making a 1-1 comparison of subgroups between the two groups, and not an ANOVA. Am I right? I will highly appreciate any insight. Thank you in advance!

$\endgroup$
  • 1
    $\begingroup$ Any time you do multiple tests and don't correct for it, you have an increased chance of a Type I error. $\endgroup$ – David Lane Mar 7 '17 at 20:44
  • 1
    $\begingroup$ @DavidLane Thanks for your comment. Here, I am doing multiple tests, but they are all 1-1. I am only comparing $v1$ of group $A$ with $v1$ of group $B$, not with $v2$, and/or $v3$ of group $A$ and/or $B$. Will multiple comparison still be an issue here? $\endgroup$ – curiousmind Mar 7 '17 at 20:53
  • 1
    $\begingroup$ The key is multiple tests. It doesn't really matter what the tests are. $\endgroup$ – David Lane Mar 7 '17 at 21:16
2
$\begingroup$

Yes.

Assuming all the null hypotheses are true, the possibility that at least one of your $t$-tests will come out significant at a probability greater than $α$ remains, and for the usual reasons. This happens in pretty much any circumstance where you run more than one hypothesis test; whether people use a multiple-testing correction of some kind seems to mostly be a matter of tradition.

$\endgroup$
  • $\begingroup$ It is not a matter of tradition. It is a necessity! $\endgroup$ – Michael Chernick Mar 7 '17 at 21:06
  • 1
    $\begingroup$ @Kodiologist wow! This is some new knowledge to me! Thank you! I tended to think multiple comparisons are an issue if we do an ANOVA since we are comparing one variable with all others (for $ABC$ we are comparing $AB$, $AC$, and $BC$). Is there any test you recommend for the multiple comparisons? Or should I adjust my $\alpha$? I would like my overall $\alpha$ to be 0.05. $\endgroup$ – curiousmind Mar 7 '17 at 21:15
  • 1
    $\begingroup$ @MichaelChernick I'm not aware of any particular necessity that separates the cases in which researchers run more than one significance test in a single study and do or don't use a multiple-testing correction. $\endgroup$ – Kodiologist Mar 8 '17 at 0:31
  • $\begingroup$ @curiousmind Choosing a multiple-testing correction is practically a field unto itself and I don't know much about it, but with only 3 tests, Holm-Bonferroni seems like a reasonable choice. You probably can't get your overall $α$ for at least one Type 1 error to be exactly .05, but you can at least limit it to .05. $\endgroup$ – Kodiologist Mar 8 '17 at 0:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.