2
$\begingroup$

I have 3 different DV that I try to model with 3 distinct models (linear mixed models) using the same set of IV. I found that the DV that I have the least amount of data for also has the lowest number of IV's in the final model, while the DV that has the largest amount of data has a larger number of remaining IV (including interactions). I tend to believe that this might be an artifact of data-availability. Might that be the case? And if so: Is that a serious problem when reporting these results in peer reviewed journal?

I know that many argue against stepwise reduction procedures, however my feeling is that (at least in my case) the selection mostly comes to the same final models irrespective of minor changes in the inital model. Are the constraints against model selection mostly valid in case of large numbers of IV or is it even in case of a relatively small number of initial IVs that there might be completely different results?

$\endgroup$
6
$\begingroup$

Yes. This is another argument against stepwise. Since it uses p-values, it will find more complex models when there is more data, even if the data is generated from the same underlying process.

This is true of all automatic methods, though (at least, all I can think of).

It's really another argument for using effect sizes, your substantive knowledge and your brain to figure out models.

| cite | improve this answer | |
$\endgroup$
  • 1
    $\begingroup$ In my case I used AIC, as I don't trust the p-values that much $\endgroup$ – Frize Oct 1 '13 at 11:19
  • 1
    $\begingroup$ OK but AIC is based on likelihood and likelihood also changes with N. $\endgroup$ – Peter Flom Oct 1 '13 at 11:23
  • $\begingroup$ Using my brain is really the last option I want to go for ;)...No seriously: is there a way that I can make use of a priori knowledge and then test model fit? I tried comparing residual plots but they all look somehow similar almost irrespective of the IVs that are considered $\endgroup$ – Frize Oct 1 '13 at 12:13
  • 1
    $\begingroup$ You can compare AIC among models that are reasonable. $\endgroup$ – Peter Flom Oct 1 '13 at 17:26
3
$\begingroup$

If you increase the sample size enough, the most complex model in your candidate set will have the lowest AIC (if it contains the others as special cases). The AIC is intended to pick the model with best predictive accuracy from a candidate set of models that are all wrong—i.e. approximations to the infinite-dimensional "model" that is reality. So in this context "too big" a model is one that over-reaches & ends up adding noise, for a given sample size. For a selection rule that converges on the least-complex true model among your candidate set as sample size increases, use the Bayesian information criterion (BIC)—if you think that the truth is simple enough to be among the models you're considering.

| cite | improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.