2
$\begingroup$

I have a 2 (time) x 3 (levels of difficulty) repeated measures design. My repeated measures ANOVA yielded a significant main effect of time, but no interaction. From this I can claim that there is a significant difference at each time point for all 3 levels of difficulty.

However, if I peek at the simple main effect (e.g., level 1 difficulty at each time) using a paired t test, the difference is non-significant at the .05 level (but is close). I know I am not supposed to do this, but can someone please explain how there can be a significant main effect, but non-significant simple main effects?

Improve this question
$\endgroup$
3
  • $\begingroup$ Was the design Latin Square? Or did everybody progress from easy to intermediate to hard? $\endgroup$
    – AdamO
    May 13, 2018 at 12:45
  • 1
    $\begingroup$ Because you are violating a principle of statistics. Spend a lot of time thinking about the model, state it statistically, fit it, check its robustness, and interpret parameters in the fully model context. Don't use pre-testing or model selection. Form interpretable contrasts that take interactions into account. Hesitate to remove interactions from the model unless a global interaction test has a huge p-value. Even then a Bayesian model for the interactions would be better. $\endgroup$
    – Frank Harrell
    May 13, 2018 at 12:55
  • $\begingroup$ what is the theme of your study ? Variables ? what is the purpose of paired t-test ? Please state your objectives and summary of data. $\endgroup$
    – user10619
    May 25, 2018 at 13:32

1 Answer 1

Reset to default
2
$\begingroup$

It's a sample size thing. The bigger the sample size the smaller the errors and therefore the higher likelihood of a positive finding (if one truly exists). As a demonstration only, instead of going back and doing the t-test for each of the individual levels, do a t-test that uses all of the levels at once (time1 including all of the values for all of the three difficulties vs. time2 including all of the values for all of the three difficulties). This will almost certainly be significant (unless there was an effect due to difficulty - which it seems like there wasn't).

Improve this answer
$\endgroup$
1
  • $\begingroup$ This is a likely explanation in my mind. Another thing to offer up is that OP's conclusion of no interaction could be a Type II error. $\endgroup$
    – AdamO
    May 13, 2018 at 12:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.