I know if I have three group means, and I compare group1-group2, then group2-group3, and group1-group3, this increases the Type1 error and I should use an ANOVA instead.

What I want to compare instead is: mean1-mean2, mean3-mean4, .... mean11-mean12. The situation is: I have a single dependent variable (match proportion, aka how much the participant looks at the answer vs distraction), but I need to make sure other factors like gender, or answer type, or the target side doesn't show a main effect.

Theoretically, there is not much reason to assume interaction between these 6 variables (e.g. no reason to think male participants will like the answer appearing on the right than female participants), so I am tempted to use multiple t-tests. But I know multiple t-tests on the same population can be a problem. (?)

My question is: (1) are multiple t-tests still a problem when different means are being compared each time and (2) if this is, what is my option? A 2x5 anova?

Any help would be greatly appreciated. Thanks.

Edit: To give more information about the structure of the study, let me give an example of what my data looks like.

Subnum  Gender  EventType  Target Side  MatchProportion  
   1      F         1           L             0.9  
   2      M         1           R             0.4  
   3      M         2           L             0.2  
   4      F         2           R             0.8  

I want to compare the average Match proportion of all four subjects against chance (0.5). But I also need to exclude the possibility of there being a main effect of gender, event type, and target side (all of these factors were included for the sake of counterbalancing, and none are of theoretical interest to me).

  • 1
    $\begingroup$ You could also do a p-value adjustment for multiplicity. The Bonferroni inequality provides one conservative method but there are many others including adjustments by bootstrap or by permutations. $\endgroup$ – Michael R. Chernick Aug 21 '19 at 17:40

Your problem isn't really whether the t-tests work, the problem is in interpreting the p-values. (That's always a problem, but people don't realize that.) If I do 1000 completely independent t-tests, and 50 are statistically significant, this is likely to have occurred by chance. But what if 100 are significant?

Are these paired t-tests? If so you have 6 means that you want to compare against zero. You could do some sort of multivariate test using a mixed model (or some other approach).

I'm not sure I quite understand your data setup though, can you give a more concrete example? Are these, for example, 6 experiments?

  • $\begingroup$ This is a single experiment. My research question do my participants look at the "correct" event more than what is predicted by chance (0.5)? My DV is mean proportion looking time to the correct event. What I am worried about though, is if there is a main effect of (e.g.) gender. If I look at the looking time average to match event of my male participants, will it be significantly higher/lower from my female participants? In other words, did men do better than women (which is something that is not of interest to me, and predicted to not be true) $\endgroup$ – ChristieSato Aug 21 '19 at 17:44
  • $\begingroup$ And eventType goes up to 12? $\endgroup$ – Jeremy Miles Aug 21 '19 at 22:01
  • $\begingroup$ No, there are only 2 event types. The variables I want to look at are Gender (F/M), Event Type(TypeA/TypeB), TargetSide(Left/Right), VocabularyScore(High/Low when split by the median Vocabulary score), PracticeScore(High/Low when split by the median of their performance in the familiarization trial prior to test) and Age (Old/Young when split by the median), making it 6 variables in total. $\endgroup$ – ChristieSato Aug 21 '19 at 22:07
  • 1
    $\begingroup$ Oh, no, this information is crucial to understanding and answering your question. 1. Multiple tests is lower on the list of problems than omitted variable bias is. If more than one predictor is important, it may be that none of the single-predictor tests mean anything useful. 2. median splitting (or any other kind of splitting) is a bad idea (this problem is discussed in a number of posts already on site). 3. You should seek advice about how to analyze your data, rather than choosing a poor analysis and then asking about problems with it. $\endgroup$ – Glen_b Aug 21 '19 at 23:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.