Evaluate different mediation analysis models in R

Question

I have 3 variables which I used to conduct a mediation analysis. Since it is unclear to me whether which variable should be the (in)dependent variable and which should be mediator, I constructed 3 models. Using package mediation, I got the results which show all three models are statistically significant:

Model 1: 
Coefficients:
            Estimate Std. Error t value Pr(>|t|)    
(Intercept)  54.0422     4.7155  11.460  < 2e-16 ***
X            27.5089     9.9512   2.764  0.00747 ** 
M             0.9369     0.4571   2.050  0.04455 *  

Causal Mediation Analysis 

Nonparametric Bootstrap Confidence Intervals with the Percentile Method

               Estimate 95% CI Lower 95% CI Upper p-value    
ACME             7.2197       0.8546        17.56   0.022 *  
ADE             27.5089      11.1184        44.51  <2e-16 ***
Total Effect    34.7285      19.1319        53.34  <2e-16 ***
Prop. Mediated   0.2079       0.0292         0.54   0.022 *  
---

Model 2:    
Coefficients:
            Estimate Std. Error t value Pr(>|t|)    
(Intercept)  54.0422     4.7155  11.460  < 2e-16 ***
X             0.9369     0.4571   2.050  0.04455 *  
M            27.5089     9.9512   2.764  0.00747 ** 
Causal Mediation Analysis 

Nonparametric Bootstrap Confidence Intervals with the Percentile Method

               Estimate 95% CI Lower 95% CI Upper p-value    
ACME              0.447        0.141         0.78   0.002 ** 
ADE               0.937        0.153         1.89   0.014 *  
Total Effect      1.384        0.544         2.37  <2e-16 ***
Prop. Mediated    0.323        0.114         0.80   0.002 ** 
---

Model 3: 
Coefficients:
             Estimate Std. Error t value Pr(>|t|)   
(Intercept) -0.158881   0.096979  -1.638  0.10635   
X            0.010813   0.005477   1.974  0.05274 . 
M            0.003932   0.001423   2.764  0.00747 **

Causal Mediation Analysis 

Nonparametric Bootstrap Confidence Intervals with the Percentile Method

                Estimate 95% CI Lower 95% CI Upper p-value   
ACME            5.44e-03     1.15e-03         0.01   0.006 **
ADE             1.08e-02    -7.11e-05         0.02   0.052 . 
Total Effect    1.63e-02     7.09e-03         0.02   0.004 **
Prop. Mediated  3.35e-01     6.56e-02         0.97   0.010 **
---

By quickly looking at ACME and also the p value for M, I would say model 2 and 3 appear to be superior than model 1. Also, for model 3, the X-->Y path was significant before the mediation effect was accounted for and no longer significant after, which isn't the case for model 2. That tells me that perhaps model 3 explains the data the best. However, is that evaluation sufficient? So my question is whether there is a way to statistically evaluate/compare these three models and pick out the one that best fits the data. I thought I'd try lavaan but from what I can see I can use it to run models with multiple mediators so not what I'm trying to do here. Any advice?

Answer 1

This is not a valid way to test the causal structure of your data. Mediation requires temporal ordering of your data; without that, any mediation analysis may be invalid. Finding a structure that yields a significant path doesn't mean that structure represents the correct causal structure of your data; it just means you found an arrangement which had a significant association. This cannot be interpreted as causal, let alone as mediation.

To test mediation, you need to measure variables at three time points. If you cannot do this, what you think is the mediator could actually be the predictor. This is a substantive matter, not a statistical one. If you don't have established temporal ordering in your data, it will be extremely challenging to prove to a reader that what you have discovered is mediation and not some arbitrary association among variables.

It is possible to generate a directed acyclic graph (DAG) of your data based on a hypothetical causal structure, use a software like DAGitty to derive testable implications of your DAG, and then test those implications. If your tests reject those implications, your DAG is incorrect. This might help you determine which way the causal arrows point.

Finally, a note on lavaan. You can use lavaan to test multiple mediators, and you can use it to test one mediator. What you cannot do is causal mediation, which includes an interaction between the treatment and the mediator and allows for variable types other than continuous for the mediator and outcome. mediation can do all of this. Also note that you cannot use fit statistics to test whether a given mediation model fits well because it will likely be a saturated model which always fits the data perfectly even if it makes no substantive sense.