I have data from a psychology experiment in which instructional method was manipulated between subjects via two factors: pretraining (3 levels) and training (2 levels). I assessed participants' performance via two tests, a pretest and a posttest, so I have test accuracy as my dv with test section (pre/post) as a within-subjects factor. I also have some other factors of secondary interest. I analyzed this data using a mixed ANOVA and found a significant (predicted) 3-way interaction of pretraining, training, and test section, indicating that the relative advantages of the two training conditions, as measured by improvement from pretest to posttest, was different depending on pretraining condition.
Now I have noticed that accuracy score in answering questions DURING the training session, which I'll call "study", is also affected by the pretraining factor, i.e. one of the pretraining conditions increases study accuracy relative to the others. I'd like to test whether this effect on study accuracy can explain the subsequent effect of pretraining on the training * test section interaction. In fact I suspect it cannot, but I'd like to test my belief. How can I do this?
My first thought was to use ANCOVA, i.e. add study accuracy as a covariate to my original ANOVA and then see whether the pretrainingtrainingsection interaction is still significant. However, in another thread (here) I was advised not to use ANCOVA when my experimentally manipulated factors are correlated with the covariate, as certainly is the case here (i.e. there is an effect of pretraining on study accuracy). So, what should I do instead?
# Sample data: nsubj = 215; nsec = 2; nprob = 6 D = data.frame( subjid = rep( 1:nsubj, each=nsec*nprob ), pretrain = rep( sample( c('a','b','c'), nsubj, replace=TRUE ), each=nsec*nprob ), training = rep( sample( c('j','k'), nsubj, replace=TRUE ), each=nsec*nprob ), study = rep( sample( 1:6, nsubj, replace=TRUE ), each=nsec*nprob ), section = rep( rep( c( 'pretest', 'posttest' ), each=nprob ), nsubj ), probtype = rep( c( 'v', 'w', 'x', 'y', 'z', 'z' ), nsec*nsubj ), accuracy = sample( c( 0.0, 0.5, 1.0 ), nsubj * nsec * nprob, replace=TRUE ) ) # Model: library( afex ) ez.glm( "subjid", "accuracy", D, within=c("section","probtype"), between= c("pretrain","training"), type=3 ) # Model with covariate: ez.glm( "subjid", "accuracy", D, within=c("section","probtype"), between= c("pretrain","training"), covariate="study", type=3 )