14
$\begingroup$

My colleague and I are fitting a range of linear and nonlinear mixed effect models in R. We are asked to perform cross-validation on the fitted models so that one can verify that the effects observed are relatively generalizable. This is normally a trivial task, but in our case, we have to split the whole data into a training part and a testing part (for CV purposes) that share no common levels. For example,

The training data may be based on Groups 1,2,3,4; The fitted model is then cross-validated on Group 5.

So this creates a problem since the group-based random effects estimated on the training data do not apply to the testing data. Thus, we cannot CV the model.

Is there a relatively straightforward solution to this? Or has anyone written a package to tackle this problem yet? Any hint is welcome!

Thanks!

Improve this question
$\endgroup$
2
  • 3
    $\begingroup$ In small area estimation, you have the same problem with "out-of-sample" small areas. What is usually done is that you estimate the out-of-sample random effects by zero (their most likely value - assuming your random effects are normally distributed). Effectively you are using the "synthetic" or fixed part of the model only for prediction. $\endgroup$
    – probabilityislogic
    Commented Nov 26, 2011 at 11:28
  • $\begingroup$ probabilityislogic/ Ting Qian, I am wrestling with this problem now, and would like to see how you specified out-of-sample effects as 0. Is it possible to edit your answer here & show the R code? thanks! $\endgroup$
    – Pradeep Babu
    Commented Nov 24, 2013 at 5:52

2 Answers 2

Reset to default
6
$\begingroup$

Fang (2011) has demonstrated asymptotic equivalence between AIC applied to mixed models and leave-one-cluster-out cross validation. Possibly this would satisfy your reviewer, permitting you to simply compute AIC as an easier-to-compute approximation to what they requested?

Improve this answer
$\endgroup$
3
  • $\begingroup$ Thanks! This looks useful. We actually already computed BIC, but the reviewer wants to see cross validation results. ;-) Some of the datasets we have are relatively small. So, one can make the argument that such asymptotic behavior is not expected. But, yes, we could certainly cite Fang(2011) when we present the BIC results, since AIC and BIC are asymptotically equivalent too? $\endgroup$
    – Ting Qian
    Commented Nov 26, 2011 at 15:03
  • 2
    $\begingroup$ I do not believe that AIC and BIC are asymptotically equivalent as they attempt to answer fundamentally different questions. See: stats.stackexchange.com/questions/577/… $\endgroup$
    – Mike Lawrence
    Commented Nov 27, 2011 at 0:37
  • $\begingroup$ And here's a more detailed comparison of AIC & BIC: smr.sagepub.com/cgi/doi/10.1177/0049124103262065 $\endgroup$
    – Mike Lawrence
    Commented Nov 27, 2011 at 0:46
2
$\begingroup$

Colby and Bair (2013) had developed a cross-validation approach that can be applied to nonlinear mixed effects models. You can visit this link to learn more.

Improve this answer
$\endgroup$
1
  • 1
    $\begingroup$ Welcome to Crossvalidated. Please add more information to your answer. Maybe you can outline the most important parts of the article. $\endgroup$
    – Ferdi
    Commented Dec 28, 2016 at 12:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.