2
$\begingroup$

Here is a rather involved question. It can be summed up as follows: how can I test the effect of a potential mediating variable, independent of the effect of a possible covariate with which it is confounded?

Here is a simplified description of my study and then the analysis problem I'm facing in more detail:

I recently conducted a study investigating the effects of different instruction methods on participants' learning outcomes. Participants were exposed to one of three TRAINING conditions. I then measured two types of learning outcomes. The first measurement, SOLUTION, is a hand-coded binary factor describing how well participants could verbalize the solution method they were supposed to have learned during training. SOLUTION is mainly of interest as a predictor of the other measurement, TRANSFER. TRANSFER is a metric variable defined as performance on POSTTEST minus performance on PRETEST.

I was originally testing the following hypotheses.

  1. TRAINING condition A would promote TRANSFER better than condition B.
  2. TRAINING condition A would also promote better SOLUTION quality than B.
  3. SOLUTION would mediate the effects of TRAINING on TRANSFER.

For

  1. I did an ANOVA with TRAINING as a between-subjects factor and TRANSFER as the DV. Unfortunately the effect was not significant, so (1) was not confirmed.
  2. was confirmed, via a simple Chi-square analysis.

So my main question relates to how to test (3). Because (1) wasn't confirmed, I restate (3) as follows: SOLUTION has an effect on TRANSFER. In particular, better SOLUTION leads to higher TRANSFER.

Now I already know that the data trend indeed is in that direction. To test whether this effect is statistically reliable, I would LIKE to simply add SOLUTION as a factor to the ANOVA I used to test (1). However, there is a complication. SOLUTION is correlated with PRETEST score, which in turn is correlated with TRANSFER. So, apparent effects of SOLUTION might actually be effects of PRETEST, right? Thus, do I need to add PRETEST into the model as a covariate?

If so, I'd then be doing an ANCOVA, and I've been advised that doing so violates the assumptions of ANCOVA, namely that the covariate should NOT be correlated with the other factors, while in this case it IS correlated with one of the factors, i.e. SOLUTION. Any way around this? Alternate analysis I should be doing? Or is it not really a problem to begin with?

If it matters, PRETEST is positively correlated with SOLUTION, but negatively correlated with TRANSFER. So maybe I'd be justified in leaving PRETEST out of the analysis because, although it is confounded with SOLUTION, their effects on transfer point in opposite directions, so leaving it out would only weaken my effect. Is this reasoning valid?

I'm also wondering more generally whether it's OK for me to add SOLUTION as a factor to my original ANOVA at all, given that SOLUTION was not randomly assigned and is itself correlated with the other factor, TRAINING, as mentioned above. Is this a problem and if so, what should I do about it?

Improve this question
$\endgroup$
1
  • 1
    $\begingroup$ Even if you don't find a significant association between training and transfer, you could still have mediation, since a null global effect could also be due to a direct and indirect effect cancelling out (basically, having opposite sign, they can both be significant while their difference not significant). You should add "pretest" as a covariate. Notice, however, that it must not be affected by "training", otherwise it would be a further mediator (the sequence would become TRAINING->PRETEST->SOLUTION->TRANSFER). $\endgroup$
    – Federico Tedeschi
    May 4, 2018 at 7:10

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Browse other questions tagged or ask your own question.