0
$\begingroup$

Running some path analysis models. I have one model with 6 variables and another model with 5 variables.

The 5 variable model has an AIC = 30 and a BIC = 80, R Squared = .30

The 6 variable model has an AIC = 40 and a BIC = 110, R Squared = .40

All other fit measures are about equal, with the 6 variable model a miniscule bit better. The chi square and p value is also minimally better for the 6 variable model.

That 6th variable we added is what we're most interested in as researchers. It adds more explanatory value, but the AIC and BIC are a little bit worse.

What would be the best route to take regarding model selection?

$\endgroup$
1
$\begingroup$

Here are my opinions,hope can help:)

At first, model fit is not the most important thing when I select the SEM model. Plus, according this paper1, AIC and BIC are not the common fit measures reported in past articles. So my first opinion, it is doesn't matter if AIC and BIC are not good enough.

Secondly, from what you write, the 6th variable is what you're most interested in as researchers. I think the core variable is more necessary than model fit.

$\endgroup$
1
$\begingroup$

As already pointed in one of the answers, it all depends on your data. For some data AIC works better, for other BIC. AIC typically favours more saturated models (i.e. more covariates are better than less), while BIC favours smaller models. You could try to add to your analysis:

  1. statistical significance, i.e. testing whether $\beta_6$, 6-th variable coefficient, is significant or not (not the same as testing all 6 variables),
  2. add an out-of-sample evaluation, i.e. run a horse race between the two models, and select the winner.

Ultimately, it depends on what you are trying to achieve and on your data. Hope this helps.

$\endgroup$
4
  • $\begingroup$ What would you do after you get the p-value for $\beta_6$? $\endgroup$ – Dave May 18 at 11:42
  • $\begingroup$ Well, if it turns out to be significant, you should keep it as in general it signals that the variable is important at least for the prediction. But keep in mind that this does not imply causality, etc. $\endgroup$ – Jonas Striaukas May 18 at 12:06
  • $\begingroup$ What if it isn’t significant? $\endgroup$ – Dave May 18 at 12:09
  • $\begingroup$ Try also out-of-sample prediction evaluation. If it becomes worse, i.e. 6 variable model gives you a less accurate prediction, likely that the additional variable is not important. In this case, keep the 5 variable model. $\endgroup$ – Jonas Striaukas May 18 at 12:12
-1
$\begingroup$

(hello, this is just post of opinion hope can help

there are a lot of characteristic criteria, I personally found every criterium has its formula, "weighting" something more/less

in simple terms, some use more weight to latest residuals, some add penalization for adding too many regressors(y~x1,x2,x3,x4 .. some care about abs residuals, some squared residuals (better handling outlayers) etc

i only guess, it is linked to usage: for example, maybe i want model with higher sum of errors, but lower maximum error. or maybe (weather vs water) highest precision is needed in area around 0 Celsius.

just in my opinion, criteriums were built to solve some kind of problems, upon which formulas was constructed (wanted solutions described by mathematic relationship). maybe in original paper(s) author(s) described/explained

so maybe just describe in you research (pros and cons), after then give suggestion based upon "weights" solution need.

$\endgroup$
1
  • $\begingroup$ why I recieved downvotes again. at -5 i will delete this answer, simply because my effort is to help, but if forum system consider my post useless, there is no reason to remain unhelpful answer. there is alot of usability of understanding measure equations or build custom ones. thats how things are built $\endgroup$ – user2120 May 18 at 20:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.