Cross-validation/bootstrap validation for mixed models After an extensive search online and unfortunately not being able to find a good answer to my question, I am wondering whether it is possible to perform e.g. k-fold cross-validation or bootstrap validation on a mixed model for longitudinal data as would usually be done in R (i.e. creating folds, using predict() etc.). I did find some source for this problem in terms of a classification problem using a mixed model (see here), however is it possible to directly translate the method of e.g. cross-validation/bootstrap validation generally used for general linear models to general linear mixed models (i.e. with a continuous outcome)? Would any problems arise in terms of prediction, performance measures? This article gives some context on cross-validation with (nonlinear) mixed models, however it is not yet clear to me whether it is directly possible.
Any useful articles or ideas would be helpful, since I have not been able to find sources of information concerning this matter in the general linear mixed model context.
 A: I think CV is pretty directly possible for mixed models (not also that confidence interval estimation via bootstrap is done a lot for mixed models): I'd actually say that with a mixed model you have the decided advantage that you spell out the data structure you consider. This structure is what you also need to correctly set up the CV splitting routine in order to get independent splits.
Here's a start:

*

*I use random factors to describe sources of variance of which an application prediction is expected to have a new, never-before encountered level.
Thus, when simulating application prediction conditions during cross validation, we want the test set to be independent in all such random factors.

*

*When random factors are nested, splitting into independent subsets at the outermost factor will automatically lead to independence also for all random factors nested within.


*When random factors are crossed, independence at both (all) factors at the same time means that we get 3 subsets of data: training data comprising a subset of the levels of the crossed random factors, test data comprising a disjoint subset of the levels of all the crossed factors. And there will be a third subset now that has for some factors "training levels" but for other factors "test levels". This part of the data can neither enter the training of the particular surrogate model, nor the test since it would break independence.
Traditional cross validation concepts do not cover this. Extensions are possible (I'm working on this), but I'm not yet aware of a standard implementation I can point you to. This is possibly because with a few crossed random factors, the available training subset soon becomes rather small compared to the full data (if you do 5-fold for 3 crossed random factors, the training subsets are only about half of the full data).  At some point, it becomes more economic to instead organize a validation study that collects independent test data.




*fixed factors are exploited for prediction, and thus no independence in splitting is required. In fact, one often goes a step further and stratifies them.


*

*The classification example linked by OP is IMHO wrong in their splitting: they formulate a (1 | patient_id) random factor, but omit to account for this in their splitting routine.


*(I have not yet read the Colby paper)
