I am comparing two models using Path Analysis (pictured).

Model A explained 49% of the variance in my DV. Model B (which is the same as Model A + an additional mediator) explained 48% of the variance of the DV.

So Model A appears to be the better model - However it makes more theoretical sense to include the Mediator (M3) from Model B. Moreover, indirect effects of the predictors through this mediator are significant.

Model Fit from both models is also good, but slightly better for Model A.

I'm hoping someone might be able to provide some advice around how to interpret these findings? How is it that the indirect effects of the predictor through the new mediator is significant, but the variance doesn't change?

enter image description here

  • $\begingroup$ Did you intentionally omit the path from IV 2 to M2 in model B? If not, comparing those models is testing not just whether the effects of IV 1 on M1 and M2 are mediated through M3 but also whether there is a direct effect of IV 2 on M2. If one model is preferred, you won't be able to disentangle which change leads to the improved performance. $\endgroup$ – Noah Sep 24 '19 at 3:13
  • $\begingroup$ @Noah Thanks for your comment. Actually the path from IV2 to M2 was non-sig once M3 was added to the model so I removed it. $\endgroup$ – LSSS Sep 24 '19 at 3:22

A few thoughts...

First, one wouldn't necessarily expect the R2 of DV to change much with the addition of the mediator. The same predictors are predicting the DV with and without the mediator. In addition, it's quite possible the difference in R2 you observed is due simply to chance and so should not be interpreted as evidence in favor of one model.

Second, comparing R2s is not the best way to evaluate the fit of a multivariate model like this one. You are testing a broad theory that includes many explanatory components, not just trying to explain variability in the outcome. Model fit statistics for SEM like CFI, RMSEA, and BIC are preferable because they assess the fit of the whole model taken at once, not just the fit of one component model for one outcome.

Third, if you leave the path from IV2 to M2 in the model, the two models are nested and you can perform a likelihood ratio test to directly compare the two models. If the test is significant, then at least one of the added paths pointing to or from M3 is significantly different from zero. You can follow up by looking at the significance of individual paths. You cannot perform this test if the path from IV2 to M2 is removed, as doing so will make the models non-nested. Note that this does not test for mediation; it merely tests whether the covariance among the observed covariates is better explained by a model with M3 than a model without. It could be that just the M3 to M1 path is nonzero, which would not correspond to mediation, but this would correspond to a rejection of the null hypothesis of the likelihood ratio test. This is a test if any of the added paths are different from zero. If they are all equal to zero, then model A is equivalent to model B.

Fourth, if you're interested in testing specific mediation hypotheses, you should perform those specific tests instead of performing omnibus tests for broad model fit. You can perform a test to see if the product of the IV1 to M3 and M3 to M1 paths is different from zero and do the same thing for the product of the IV1 to M3 and M3 to M2 paths. If these are significant, then you evidence that the effects of IV1 on M1 and/or M2 pass (at least partially) through M3. The null hypothesis of each of these tests is that at least one of the component paths is equal to zero. Note that this differs from the likelihood ratio test, the null hypothesis of which is that all of the additional paths are equal to zero.

Finally, don't remove paths just because they are nonsignificant. Doing so can inappropriately change your model. Just estimate the path and report it as nonsignificant. Nonsignificant doesn't mean it's equal to zero and should be constrained as such; it just means you didn't have evidence to claim it's different from zero. There are variable selection techniques to determine which paths should be estimated and which can be constrained to be zero, but these techniques are purely exploratory and should not be used in a confirmatory fashion, as you are attempting to do when testing a specific theory.

  • $\begingroup$ Yes this makes sense and thank you for your detailed and clear response. My hypotheses relate more to whether the IV's predict the DV and the mechanisms by which this effect happens (i.e. mediation) rather than finding the best model, so it appears that it might not be necessary to evaluate the model fit comparatively. $\endgroup$ – LSSS Sep 24 '19 at 4:39
  • $\begingroup$ Additionally, my approach to these models has been exploratory in the sense that I have theoretical reason to believe that IVs will be related to the DV and that this will be mediated by M1 & M2 (and possibly M3). However research has not been done in this area before and specific predictions about relationships weren't possible. Consequently, my approach was to test a saturated version of each model, then trim the non-significant paths which from my very basic understanding of SEM is not an unusual approach. Would you recommend leaving all pathways in instead (i.e. having a saturated model)? $\endgroup$ – LSSS Sep 24 '19 at 4:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.