P values non-significant but Monte Carlo confidence interval does not contain zero for indirect effects

Question

I've run a path analysis using semTools. I'm interested to test indirect effects. The p values for all indirect effects were non-significant, but the Monte Carlo confidence interval for some of them did not contain zero.

My question is - is it appropriate to use only Monte Carlo confidence interval, but not p-values to report and interpret the results?

I did some search online but I'm afraid there was limited info. I saw this post: Non-significant p-values but CI does not include 0, but the answer talked about bootstrapping. In my study, I used multiple imputations to handle missing data, so bootstrapping was not appropriate.

Below are some codes showing the conflicting results. Because the codes for my model is long, I didn't show it below. I've shown the codes that are relevant to my question. Indirect effect 19 and 20 below had non-significant p values, but the Monte Carlo CI did not contain zero.

I saw some people saying that they would not trust the p-values because they are computed assuming the Z-statistic comes from a standard normal distribution (thus potentially favouring Monte Carlo CI). But I also saw some people saying that the p values and CI (not specific to Monte Carlo though) should be consistent with each other. If they are not consistent, I should not reject the null hypothesis. Can anyone please shed some light on this?

output <- runMI(model, data=data3c.mi.1, fun="sem", estimator = "MLR", se = "robust.huber.white")


                Estimate   Std.Err  t-value     df  P(>|t|) ci.lower ci.upper   Std.lv  Std.all
indirect19        0.118    0.074    1.593 1563.011    0.111   -0.027    0.264    0.119    0.008
indirect20        0.090    0.055    1.639 1924.600    0.101   -0.018    0.198    0.091    0.009
indirect21        0.146    0.126    1.152  714.059    0.250   -0.102    0.393    0.147    0.010
indirect22        0.112    0.097    1.147  664.112    0.252   -0.080    0.303    0.112    0.011

monteCarloCI(output, standardized = TRUE)

                           est ci.lower ci.upper
indirect19              0.119    0.001    0.297
indirect20              0.091    0.001    0.215
indirect21              0.147   -0.059    0.393
indirect22              0.112   -0.044    0.303

Answer 1

P-values and confidence interval extremes are functions of the data and the statistical models used in their calculation. That means that it is wrong to expect that a p-value from one method will 'agree' with the confidence interval from another.

Your results suggest that you have weak evidence that the null hypothesis is false according to both methods. Slightly weaker according to the model that you used to generate p-values than according to the model that gave you intervals.

Answer 2

Make a plot, and see that these confidence intervals are very much like each other.

So the two features 'indirect19' and 'indirect20' are border cases. This makes the question not only about which method is best (and all sort of details about both methods which are estimations and not exact, thus likely to slightly differ), but it also opens up the can of worms about p-values at the border of a predefined significance level (e.g. Is the exact value of a 'p-value' meaningless? or Is it wrong to refer to results as "nearly" or "somewhat" significant?).

The appropriateness of reporting only one set of (convenient) p-values... In this specific case it doesn't really have much influence, but that practice is generally not very good. It can be considered p-hacking. People perform some experiment and, when the result is not very accurate, instead of gathering more data to be more sure, they try out many different statistical tricks to make the results look better than they really are.

It is better to just report everything that you did and represent the data for what it is, without cherry picking regarding p-values. If it is not very significant then it is just not significant. In the end, what matters is actually the effect size, and significance is just a measure of the precision in the experiment.

Also, be clear why you performed multiple tests. If you were not sure about what method is appropriate, then do not just report only a single one when the conclusions about significance suit you better. The difference in significance is not what makes the one method better than the other. If you couldn't decide which statistical method to use before doing the analysis, then you can neither after having performed the calculations. (you might figure out too later, after seeing the results that one method happened to be more powerful, but it is a slippery slope to decide afterwards based on the observed power)

Answer 3

The normal-theory CIs in summary() are consistent with the normal-theory p values in summary(). The monteCarloCI() method makes less restrictive assumptions, so it is more defensible (see the reference on the help page). Not that it honestly makes a big practical difference. The CIs that do not contain 0 do contain .001 (which is nearly 0), so you can't reject that null hypotheses either.

FYI, you should use estimator="MLM". It is more efficient than MLR, which is only advantageous when you have missing data and use FIML estimation. The whole point of multiple imputation is to be able to use complete-data methods.