1
$\begingroup$

I've conducted a 4 way repeated measures ANOVA and have a number of significant interactions to analyse post-hoc. I've been informed by my supervisor to conduct multiple paired t-tests and then apply a confidence interval adjustment.

My question is: Which values should the t-tests be conducted on? Is it the case that all simple main effects should be compared at all levels of each other? This seems unintuitive as significant differences at some levels wouldn't be of much interest from a theoretical point of view.

$\endgroup$
1
$\begingroup$

From wiki:

Rather than testing each hypothesis at the $\alpha/m$ level, the hypotheses may be tested at any other combination of levels that add up to $\alpha$, provided that the level of each test is determined before looking at the data.[7] For example, for two hypothesis tests, an overall $\alpha$ of $.05$ could be maintained by conducting one test at $.04$ and the other at $.01$.

Your intuition is correct, you only need to conduct difference of means tests (with a bonferroni correction) for levels that you are interested in comparing in advance. Choosing the comparisons you want to make after looking at preliminary results is a form of p-hacking which could bias your results.

| cite | improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.