I am comparing two groups of people: Group $A$ and group $B$. For each group, I measured three variables: $v1$, $v2$ and $v3$, all are numeric. My intention is to compare the two groups based on the overall score ($v= v1+v2+v3$), $H_{0}: \mu_{vA} = \mu_{vB}$ and also by the subscore ($H_{0}: \mu_{v1A} = \mu_{v1B}, H_{0}: \mu_{v2A} = \mu_{v2B}, H_{0}: \mu_{v3A} = \mu_{v3B}$). I will be doing a two sample independent T-Test for testing these hypotheses. To me it seemed like these individual comparisons of subscore will not pose a problem of multiple comparisons since I am making a 1-1 comparison of subgroups between the two groups, and not an ANOVA. Am I right? I will highly appreciate any insight. Thank you in advance!
2
$\begingroup$
$\endgroup$
-
1$\begingroup$ Any time you do multiple tests and don't correct for it, you have an increased chance of a Type I error. $\endgroup$ – David Lane Mar 7 '17 at 20:44
-
1$\begingroup$ @DavidLane Thanks for your comment. Here, I am doing multiple tests, but they are all 1-1. I am only comparing $v1$ of group $A$ with $v1$ of group $B$, not with $v2$, and/or $v3$ of group $A$ and/or $B$. Will multiple comparison still be an issue here? $\endgroup$ – curiousmind Mar 7 '17 at 20:53
-
1$\begingroup$ The key is multiple tests. It doesn't really matter what the tests are. $\endgroup$ – David Lane Mar 7 '17 at 21:16
2
$\begingroup$
$\endgroup$
Yes.
Assuming all the null hypotheses are true, the possibility that at least one of your $t$-tests will come out significant at a probability greater than $α$ remains, and for the usual reasons. This happens in pretty much any circumstance where you run more than one hypothesis test; whether people use a multiple-testing correction of some kind seems to mostly be a matter of tradition.
answered Mar 7 '17 at 21:04
-
$\begingroup$ It is not a matter of tradition. It is a necessity! $\endgroup$ – Michael Chernick Mar 7 '17 at 21:06
-
1$\begingroup$ @Kodiologist wow! This is some new knowledge to me! Thank you! I tended to think multiple comparisons are an issue if we do an ANOVA since we are comparing one variable with all others (for $ABC$ we are comparing $AB$, $AC$, and $BC$). Is there any test you recommend for the multiple comparisons? Or should I adjust my $\alpha$? I would like my overall $\alpha$ to be 0.05. $\endgroup$ – curiousmind Mar 7 '17 at 21:15
-
1$\begingroup$ @MichaelChernick I'm not aware of any particular necessity that separates the cases in which researchers run more than one significance test in a single study and do or don't use a multiple-testing correction. $\endgroup$ – Kodiologist Mar 8 '17 at 0:31
-
$\begingroup$ @curiousmind Choosing a multiple-testing correction is practically a field unto itself and I don't know much about it, but with only 3 tests, Holm-Bonferroni seems like a reasonable choice. You probably can't get your overall $α$ for at least one Type 1 error to be exactly .05, but you can at least limit it to .05. $\endgroup$ – Kodiologist Mar 8 '17 at 0:34
