27
$\begingroup$

How are (linear) mixed effects models normally compared against each other? I know likelihood ratio tests can be used, but this doesn't work if one model is not a 'subset' of the other correct?

Is the estimation of the models df always straightforward? Number of fixed effects + number of variance components estimated? Do we ignore the random effects estimates?

What about validation? My first thought is cross validation, but random folds might not work given the structure of the data. Is a methodology of 'leave one subject/cluster out' appropriate? What about leave one observation out?

Mallows Cp can be interpreted as an estimate of the models prediction error. Model selection via AIC attempts to minimize the prediction error (So Cp and AIC should pick the same model if the errors are Gaussian I believe). Does this mean AIC or Cp can be used to pick an 'optimal' linear mixed effects model from a collection of some un-nested models in terms of prediction error? (provided they are fit on the same data) Is BIC still more likely to pick the 'true' model amongst candidates?

I am also under the impression that when comparing mixed effects models via AIC or BIC we only count the fixed effects as 'parameters' in the calculation, not the actual models df.

Is there any good literature on these topics? Is it worth investigating cAIC or mAIC? Do they have specific application outside of AIC?

Improve this question
$\endgroup$
2
  • 2
    $\begingroup$ What do you mean by application of cAIC or mAIC "outside of AIC"? DIC is a widely-used measure of predictive accuracy that you could investigate, that tries to penalize by the "effective" number of parameters included in the multilevel model. $\endgroup$
    – guest
    Feb 28, 2012 at 7:55
  • $\begingroup$ @guest I mean, do they have specific use, say for particular types of models? I will check out DIC. Thank you. $\endgroup$
    – dcl
    Feb 28, 2012 at 8:11

2 Answers 2

Reset to default
13
+25
$\begingroup$

The main problem on model selection in mixed models is to define the degrees of freedom (df) of a model, truly. To compute df of a mixed model, one has to define the number of estimated parameters including fixed and random effects. And this is not straightforward. This paper by Jiming Jiang and others (2008) entitled "Fence methods for mixed model selection" could be applied in such situations. A new related work is this one by Greven, S. & Kneib, T. (2010) entitled "On the behavior of marginal and conditional AIC in linear mixed models". Hope this could be helpful.

Improve this answer
$\endgroup$
1
  • 2
    $\begingroup$ I will check those papers out. Cheers. $\endgroup$
    – dcl
    Mar 7, 2012 at 9:26
7
$\begingroup$

One way to compare models (whether mixed or otherwise) is to plot results. Suppose you havae model A and model B; produce the fitted values from each and graph them against each other in a scatter plot. If the values are very similar (using your judgement as to whether they are) choose the simpler model. Another idea is to find the differences between the fitted values and graph these against the independent values; you can also make a density plot of the differences. In general, I am a proponent of not using statistical tests to compare models (although AIC and its variants certainly have virtues) but rather using judgement. Of course, this has the (dis)advantage of not giving precise answers.

Improve this answer
$\endgroup$
4
  • $\begingroup$ What you are describing, just is to compare models when the main aim is their predictive ability. Also, graphical results could be very informative to guide one which models can be useful, but, generally, they are not fully formal scientific results. $\endgroup$
    – hbaghishani
    Mar 2, 2012 at 5:53
  • 4
    $\begingroup$ Hi @hbaghishani; I'll just quote Tukey "Far better an approximate answer to the right question, which is often vague, than an exact answer to the wrong question, which can always be made precise." :-). It's not completely apropos here, but it is at least partially on target $\endgroup$
    – Peter Flom
    Mar 2, 2012 at 10:46
  • 1
    $\begingroup$ I typically to do plots such as you describe while model building. But I was indeed looking for a more 'mathematical' method. Cheers $\endgroup$
    – dcl
    Mar 7, 2012 at 9:28
  • $\begingroup$ If comparing different models based on predictive performance, my understanding is that predicted values for mixed models with and without random effects should be identical (i.e., regression coefficients will be unbiased in models with and without random effects, only the standard errors change). $\endgroup$
    – RobertF
    Nov 26, 2013 at 16:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.