11
$\begingroup$

Given that we favour parsimony in science, should we remove nonsignificant paths that are nonsignificant if they do not result in significantly worse model fit?

After we remove the nonsignificant paths, and other paths in turn become nonsignificant, do we continue to remove paths until all the paths are statistically significant (if they do not result in worse model fit of course)?

If not, how then do we decide which path to remove?

Improve this question
$\endgroup$

4 Answers 4

Reset to default
17
$\begingroup$

An SEM analysis is really just a series of more-or-less related regression models, so the general question here is really just "once you have run a regression model, is it a good idea to drop the non-significant variables and re run it?" And the general consensus among most statisticians is that this is not a good idea because running different versions of the same model undermines the logic of significance tests, increasing the chance that you will get a significant result just due to chance. In extreme cases this can serve as a form of (unintentional) "p hacking." Here is the obligatory xkcd cartoon that everyone cites to explain why running the same model over and over again causes this problem.

The best approach is usually to use theory and background knowledge do specify a single model (in this case a single set of models as part of an SEM) that reflect your hypotheses about how variables should be correlated. Then you run that one model, and report the results, no matter what they are. Non-significant results are still results (sometime they are really interesting results!) and should be reported. If you remove non-significant variables and re-run the model you are not only hiding those results, but hiding the fact that you ran more analyses than the one you presented.

Improve this answer
$\endgroup$
1
  • 6
    $\begingroup$ This is a great answer. It may be defensible to fit more than one model if you specify them a priori, perhaps based on different hypotheses or theories, and compare them. But you wouldn't want to compare more than three or four well-chosen models in that case. Definitely agree that fitting multiple models based on significance of terms in a previous model is problematic. $\endgroup$
    – qdread
    Commented Oct 31 at 13:07
12
$\begingroup$

I don't think that this is good practice given that you would "gain" additional model degrees of freedom after you've already looked at the data/model results. So your fit (chi-square) would likely look "too good" after this kind of data-driven trimming of non-significant effects.

Improve this answer
$\endgroup$
0
10
$\begingroup$

Answer

Sadly, this is actually a very common practice, but as Graham and Christian both note, this is not defensible. The idea behind structural equation modeling is to fit a model and then try to "knock it down" to see if it holds up to your claims. If that foundation cracks, then trying to sort and glue together pieces post-hoc without having pre-supposed those relationships verges on unscientific (from Hayduk, 2014, p.4 on similar issues with modification indices):

When a model fails, researchers routinely examine the modification indices to see whether introduction of specific model coefficients would improve model fit. A coefficient that significantly improves model fit will, if freed, also result in a statistically significant coefficient estimate. Hence, scanning the modification indices for coefficients capable of significantly improving model fit constitutes a fishing expedition. Selective inclusion of data-prompted coefficients increases concern for both coefficient-fishing and model-fishing.

There is beauty in reporting failed models. It tells the rest of the world where not to venture, or it may make others question why this failed in the first place. When a field is dominated by models that always work, then we run the risk of false confidence. As they say, the social sciences need to get tired of always winning (Haeffel, 2022)

References

Improve this answer
$\endgroup$
5
$\begingroup$

All of these answers are excellent. I just wanted to take the time to reiterate the difference between local and global fit in SEM. The OP is sort of asking about how local parameter estimates might be used in changing how the global model is specified. That would, in turn, affect the new measure of global fit, as Christian mentioned. If using the $\chi^{2}$ test of fit, the results may or may not be comparable between models, depending upon if they're nested or not. In a simple model, deleting one path SHOULD result in a model nested in the one with the path (we're setting the path coefficient to 0), but I'm sure there are examples I can't think of that might result in a non-nested model.

So my answer echoes the others such that

  1. Using p-values for model selection has a known bias regardless of the type of regression analysis one is doing
  2. A priori models, specified using substantive theory, can (and probably should) be compared, either with nested model comparison tests (usually likelihood-ratio based), or with other metrics such as AIC
  3. Parsimony is ONE way to choose between models, but it shouldn't be a default strategy, and it doesn't provide any rationale to select variables after p-values are seen

As one of the other answers mentions, this is consistent with good model selection practice across the board.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.