I'm planning to contrast two independent groups of participants on a particular performance measure in order to determine which of two theoretical models explains their behaviour. I will calculate a value that we hypothesize will have a mean near 0 according to Model 1, and a mean significantly different from 0 according to Model 2. As the goal is to determine, per group, which model the data best matches, we will be using 1-sample t-tests for the two groups.

My question is, would I still correct for multiple comparisons given that the test is being run on two independent groups? There is also no statistical comparison between the groups.

In summary, we will run a 1-sample t-test for group 1, and a 1-sample t-test for group 2. Should we adjust the alpha? And which correction do you recommend?

  • $\begingroup$ Let's call your two 1-sample tests, test A and B. Think about the logical relation between A and B test. Do they depend on each other? Do you require an AND relation between them, such as A being significant AND B not significant? $\endgroup$ – NULL Aug 18 '17 at 12:46
  • $\begingroup$ I would say that they do not statistically depend on each other. For the purpose of the study, they are of course related. One is a clinical population, the other is a typical/healthy population. But the two tests do not depend on each other, and there are no shared data-points between them. Is this what you're getting at @NULL? $\endgroup$ – James Trujillo Aug 21 '17 at 10:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.