# K-fold cross validation for GLMM?

hope all of you are safe during these pandemic scenario.

I would describe my case so you can get a better understatement of my question.

I work with a wild population of a bird species in which we can find two different types of individuals: those with a subordinate status called floaters (birds with no territory which is secretively moving inside the population) and those with a high status called territorials (birds that defend a territory, a partner and a nest). Our fieldwork procedure let us distinguish through their behavior between these two types of individuals and our currently question is if there are any morphological differences between floaters and territorial individuals.

For this aim, we have 7 cohorts of 1-year adults and its morphological data (weight, tarsus length (both to measure condition), length of ornamental feathers, amount spotted plumage... etc) to compare between floaters and territorials (F vs. T).

And I've built the following model after exploring and clearing up my data set: a logistic regression using lme4 with year as a random factor. After getting the results on the summary, I've checked the residuals using the package DHARMa and everything is ok.

logistic <- glmer(status ~ body_condition + plumage + feather_length + (1|year)
data = data, family = "binomial")

summary(logistic)

Generalized linear mixed model fit by maximum likelihood (Laplace Approximation) ['glmerMod']
Family: binomial  ( logit )
Formula: status ~ body_condition + mot_suma + feather_length + (1 | capture_year)
Data: data2

AIC      BIC   logLik deviance df.resid
239.4    255.2   -114.7    229.4      171

Scaled residuals:
Min      1Q  Median      3Q     Max
-1.8357 -0.7930 -0.5998  1.0149  2.2089

Random effects:
Groups       Name        Variance Std.Dev.
capture_year (Intercept) 0.1558   0.3947
Number of obs: 176, groups:  capture_year, 6

Fixed effects:
Estimate Std. Error z value Pr(>|z|)
(Intercept)    -1.89970    2.01301  -0.944  0.34532
body_condition -0.51050    0.18749  -2.723  0.00647 **
mot_suma       -0.46016    1.10950  -0.415  0.67833
feather_length  0.04145    0.05569   0.744  0.45668
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Correlation of Fixed Effects:
(Intr) condtn mot_sm
body_condi   0.179
mot_suma    -0.081 -0.057
fethr_lngth -0.991 -0.177  0.064


I am new to GLMMs and I was wondering if after checking the model residuals it would be necessary to do any other type of validation like K-fold cross validation to demonstrate that the built model is actually right.

I've spent the last days searching for ways to perform k-fold CV in R for GLMM but I've only found how to perform it with a glm, not a glmm. I could use some help right now because I feel quite lost!

• k-fold validation doesn't tell you the model is right. It is used normally to check what would be good hyper parameter settings, without over fitting the model. That said, what does cross-validation on a glm tell you? May 17 '20 at 12:21
• i'd would say that cv tells if a glm could be generalised for assesing the status of new gathered birds in a future, lets say I want to know if an individual is floater or territorial counting only with some variables that help discriminate the status, doesn't it? As I said I'm learning this for the first time on my own, so any clarification is welcome! Thank you :-)
– Iraida
May 17 '20 at 12:51
• If it is the future, I can see you have year as a random effect. Not very clear how is that going to be used without involving the data you have now. I think your aim for doing cross validation is not very well defined May 17 '20 at 13:20
• This is a statistical question, you can post it on cross-validated, but yeah you can use caret etc to split your dataset and do the CV. I am just not so sure about the interpretation, hence I did not post an answer May 17 '20 at 13:20

As Stupidwolf already commented, you cannot use cross validation (or any other resampling validation) to show that your model is right in the sense that it is correctly specified.

IMHO it is actually rather the other way round: you need the model specification (or data structure) in order to correctly set up your resampling procedure.
In particular, the splitting procedure needs to take the structure and relationship between the random factors in your model into account in order to test levels of the random factors that are independent from the levels that occur in training. In your case, this means splitting not only by case but by year. If cases are nested within year (as the residuals in your specification), that will automatically achieve independence for both.

So: knowledge about appliaction, data generation process and statistical design of experiments are needed together to arrive at a correct validation experiment.

But cross validation cannot tell you e.g.

• whether/which factors should be random and which should be fixed, or
• whether your data should be modeled in a paired, blockwise, ... fashion