If there were some path coefficients that were significant but the whole model shows poor fit, can I still use the result from the significant path?
By extension, if the model evidenced poor fit in structural equation modelling (SEM), would it be appropriate to use Hayes' PROCESS macro to analyse the data as it doesn't provide fit indices like in SEM?
You pose some interesting questions here. With respect to global versus local $p$ values...if we were to put statistical significance of path models in a hierarchy, then our first assessment should be whether or not our global/local fit measures are good. Our $p$ values become a bit meaningless if we have terrible fit because the model as a whole may be trash. For example, having a bunch of manifest variables which are modeled as uni-dimensional when they should be multi-dimensional may yield some "satisfactory" $p$ values, but their interpretation may be completely pointless if we know a multi-dimensional structure is far more plausible. Jeremy Miles provides a great example of how mis-specification amid "healthy" coefficients can also be dangerous. I simulate such a model in R below, where the true population model is a partial mediation between $X$, $M$ and $Y$, but the model omits the path from $X$ to $Y$:
#### Simulate Data ####
library(lavaan)
set.seed(123)
n <- 10000
x <- rnorm(n)
m <- .5*x + rnorm(n)
y <- .5*m + .5*x + rnorm(n)
df <- data.frame(x,m,y)
#### Specify Model ####
model <- "
y ~ m
m ~ x
"
#### Fit Model ####
fit <- sem(model,df)
summary(fit, fit.measures = T)
The model summary shows that the path coefficients are strong enough and we have statistical significance, but the global fit indices are wack.
lavaan 0.6-19 ended normally after 1 iteration
Estimator ML
Optimization method NLMINB
Number of model parameters 4
Number of observations 10000
Model Test User Model:
Test statistic 1743.173
Degrees of freedom 1
P-value (Chi-square) 0.000
Model Test Baseline Model:
Test statistic 8184.060
Degrees of freedom 3
P-value 0.000
User Model versus Baseline Model:
Comparative Fit Index (CFI) 0.787
Tucker-Lewis Index (TLI) 0.361
Loglikelihood and Information Criteria:
Loglikelihood user model (H0) -29259.957
Loglikelihood unrestricted model (H1) -28388.371
Akaike (AIC) 58527.915
Bayesian (BIC) 58556.756
Sample-size adjusted Bayesian (SABIC) 58544.045
Root Mean Square Error of Approximation:
RMSEA 0.417
90 Percent confidence interval - lower 0.401
90 Percent confidence interval - upper 0.434
P-value H_0: RMSEA <= 0.050 0.000
P-value H_0: RMSEA >= 0.080 1.000
Standardized Root Mean Square Residual:
SRMR 0.118
Parameter Estimates:
Standard errors Standard
Information Expected
Information saturated (h1) model Structured
Regressions:
Estimate Std.Err z-value P(>|z|)
y ~
m 0.697 0.010 72.392 0.000
m ~
x 0.506 0.010 49.946 0.000
Variances:
Estimate Std.Err z-value P(>|z|)
.y 1.176 0.017 70.711 0.000
.m 1.014 0.014 70.711 0.000
If we consider models with latent variables, we also run into the following issue (from Kline, 2023, p.24):
Standard errors for the effects of latent variables are estimated by the computer, and those standard errors are the denominators of significance tests for those effects. The value of the standard error could change if, say, a different estimation method is used or sometimes even across different software packages for the same analysis and data. Thus, it can happen that an effect for a latent variable is “significant” in one analysis but is “not significant” in another analysis with a different estimator or computer tool.
So this implies that the significance of each path is highly dependent on how the model is organized, so that they are not independent in the way a more standard linear regression's coefficients would be.
As for your second question, I'm assuming you mean that you are moving from a latent variable mediation model in a SEM and moving to a mediation model with only manifest variables (this doesn't require PROCESS but is often estimated with it). This would depend on your purpose. If for example a single indicator is a much more plausible and noiseless predictor of the mediator/outcome, then it is sensible to use this instead of something that has fifty manifests that all construct a pretty terrible latent variable.
The key is to not just cherry pick variables to get the right model/path fits. It should be guided heavily by theoretical and empirical justifications, along with reporting of your "failed" models.
Kline, R. B. (2023). Principles and practice of structural equation modeling (5th ed.). The Guilford Press.
