Lets say I have 2 factors and fully balanced groups:

  1. Between-subjects: condition (0, 1)
  2. Within-subjects: time (time1, time2)

I run a 2x2 between-within ANOVA and find a significant interaction. I proceed to test the simple main effects. I'm primarily interested in whether condition0 and condition1 are different from each other at time2

Is that simple main effect test the same as if I just used the time2 data and ran an independent samples t-test between the two conditions?

For reference, these are the two tests I'm referring to in SPSS syntax:

GLM time1 time2 BY condition
  /WSFACTOR=time 2 Polynomial 
  /EMMEANS=TABLES(time*condition) compare(condition)

T-TEST GROUPS=condition(0 1)

I was always of the understanding that the tests are different and should give different results, and the answer here implies that simple main effect tests are not the same as separate t-tests. However, I created some normally distributed test data, ran the code above, and all of my results are identical

Should I expect to always see the same results (p value, degrees of freedom, standard error, CI values etc)? More importantly, under what conditions would you expect the two sets of results to differ?

If they are always expected to be the same, why would you ever run the (arguably) more complicated simple main effects test and not just use t-tests every time (assuming an interaction has been found)?

  • $\begingroup$ test the difference at time 2 based on ANOVA and t-test using only time 2 data are different, especially on df. I am not familiar with SPSS, so can not figure out how the random effect was treated in your code. If you can write the math model, especially random effect part, I can give you detailed explanation. $\endgroup$ – user158565 Nov 18 '18 at 5:20

They are not equivalent, and the reason boils down to the error term. When you conduct a t-test, you compute your error term based only on the subset of data being compared. However when you conduct the ANOVA, you’re using a different error term taking all data into account. From a strict statistical standpoint, if you can model the entire dataset, you should, and that error term should be used.

  • $\begingroup$ I can see how this is true when considering the omnibus 2way ANOVA, but is the error term for a simple main effect test also different from the error term in a t-test? What is the error term in each case? Also, how would that explain me getting the same results on my test data? $\endgroup$ – Simon Nov 19 '18 at 23:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.