5
$\begingroup$

I have just created my first mediation model using sem() with the lavaan package in R. I am using a bootstrapping with 5000 resamples and BCA to calculate the confidence intervals at a 0.9 level.

However, now I face difficulties in interpreting my results, as I get non-significant p-values (>.1) for the indirect effects, while the confidence intervals do not include 0. How can or should I interpret such results? Are the CI sufficient to talk about a indirect effect?

I have already researched the web and also this forum but haven't found anything that helps me with the given situation. The only thing I found was this post: https://stats.stackexchange.com/a/302949/353584

My Code:

mediation_model <- '
#Regressions Meditation BC -> Aff -> Exp
v_13 ~ a1 * BC 
v_36 ~ a2 * v_13 + b1 * BC + ExpMan
#Regressions Meditation Exp -> Disco -> Rating
v_3 ~ a3 * v_36 + e1 * BC + ExpMan + v_30 + PerfMan + e2 * v_13
v_6 ~ a4 * v_36 + a5 * v_3 + e3 * BC + ExpMan + v_30 + PerfMan + e4 * v_13 + a6 * v_5
#Regressions Mediation Exp -> Disco -> Sat
v_5 ~ b2 * v_36 +  a7 * v_3 + e5 * BC + ExpMan + v_30 + PerfMan + e6 * v_13 
ACME1 := a1 * a2
ACME2 := a3 * a5
ACME3 := a3 * a7
ACME4 := b2 * a6
indESR := b2 * a6
indEDSR := a3 * a7 * a6
BCind1 := a1*a2*a3*a7*a6
BCind2 := a1*a2*b2*a6
BCind3 := a1*a2*a4
BCind4 := a1*a2*a3*a5
BCind5 := b1*a3*a7*a6
'
fit <- sem(model = mediation_model, data = data_all, se = "boot", bootstrap = 5000)

summary(fit, fit.measures = T, modindices = F)
parameterEstimates(fit, boot.ci.type = "bca.simple", level = 0.9)

Results look like:

       lhs op            rhs   label    est    se      z pvalue ci.lower ci.upper
41   ACME1 :=          a1*a2   ACME1  0.092 0.058  1.578  0.115    0.004    0.198
42   ACME2 :=          a3*a5   ACME2 -0.135 0.043 -3.168  0.002   -0.225   -0.079
43   ACME3 :=          a3*a7   ACME3 -0.311 0.092 -3.378  0.001   -0.480   -0.175
44   ACME4 :=          b2*a6   ACME4  0.069 0.043  1.588  0.112    0.008    0.148
47  BCind1 := a1*a2*a3*a7*a6  BCind1 -0.008 0.007 -1.283  0.199   -0.024   -0.001
Improve this question
$\endgroup$

1 Answer 1

Reset to default
7
$\begingroup$

You should only expect confidence intervals and p-values to align in tests of significance when the same method is used to compute both. In this case, the p-value is computed using a Z-statistic which is computed as the estimate divided by the bootstrap standard error. The confidence interval doesn't use the standard error at all; it is an adjustment to the percentile bootstrap.

I would not trust the p-values because they are computed assuming the Z-statistic comes from a standard normal distribution, which for most mediation-related quantities is not true in finite samples. The bootstrap confidence intervals are the most accurate because they make few assumptions about the sampling distribution of the quantity of interest. I would not even report p-values in your paper and rely instead on the bootstrap confidence intervals. See for example Hayes and Scharkow (2013), who recommend this method (though the Z-test reported by lavaan is not one of the methods compared).

Improve this answer
$\endgroup$
5
  • $\begingroup$ What would you think of giving the p-value as the largest $p$ for which a $(1-p)\times 100\%$-level bootstrap confidence interval has zero as an endpoint? $\endgroup$
    – Dave
    Commented Mar 24, 2022 at 23:19
  • $\begingroup$ Thanks a lot for your answer Noah! I have found another paper supporting your claim with Hayer (2009). Perhaps you could help me with another problem of mine. While researching, some practioners added covariances between the mediators into the SEM model, others do not. Literature on this topic is (at least for me) very confusing. When exactly is it needed to include these covariances of the mediators and does it harm the model to not include them? I tried including them and my CIs got more robust, however my model fit is not given anymore as I end up with negative degrees of freedom. $\endgroup$
    – 140989
    Commented Mar 25, 2022 at 0:24
  • $\begingroup$ @Dave I can't speak to the validity of that, but I think that would be equal to twice the proportion of bootstrap estimates less than 0 (if the point estimate is greater than 0). $\endgroup$
    – Noah
    Commented Mar 25, 2022 at 4:19
  • $\begingroup$ @140989 You should ask a new question since that is separate from this one. An SEM expert on the site may be able to give you a better answer. $\endgroup$
    – Noah
    Commented Mar 25, 2022 at 4:20
  • $\begingroup$ @Noah Thank you! I appreciate it $\endgroup$
    – 140989
    Commented Mar 25, 2022 at 9:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.