I recently ran a k-fold cross validation on a data set/model that I was interested in evaluating the performance of. In doing so, I received a value of 0.46. I'm assuming this low value indicates poor model predictability but wasn't 100% sure. Thanks for the help!

enter image description here

results_full <- glmer(R0A1~MP_Scaled+MPHW_Scaled+HW_Scaled+YP_Scaled+AG_Scaled+Shrub_Scaled
    data=secondorder, family=binomial)
  • $\begingroup$ 0.46 what? What is your metric? Accuracy? $\endgroup$ Feb 6, 2015 at 16:44
  • $\begingroup$ 46% correct classification $\endgroup$
    – Buck2079
    Feb 6, 2015 at 17:26
  • $\begingroup$ @Buck2079 Did you find a way to perform k-fold cross validation of GLMM's in R? I am having the same problem now... Thank you in advance!! $\endgroup$
    – mto23
    Jul 13, 2016 at 10:43

1 Answer 1


It completely depends on the situation. You have to understand the context of your data or at least know how well others have performed before you. For balanced binary classification, then 46% is worse than tossing a coin. But let's say your model have identified a small group of start-up companies, where 46% increased their assets more than 10-fold the following year. Then the cost of false-positive is negligible and your model is performing incredibly well.

simple k-fold cross-validation code:

noise.factor = 2

#a data.matrix
X = replicate(vars,rnorm(obs))
true.coefs = rnorm(vars)
y = as.numeric(true.coefs %*% t(X)) + rnorm(obs) * noise.factor

#a simple kfold code
folds = split(sample(1:length(y)),1:kfold)
test.preds.matrix = foreach(i = folds,.combine=cbind) %do% {
 Data.train = data.frame(X=X[-i,],Y=y[-i])
 Data.test  = data.frame(X=X[i ,],Y=y[ i])
 lmf = lm(Y~.,Data.train)  
 test.pred = rep(0,length(y))
 test.pred[i] = predict(lmf,Data.test)
#diagnostics plot
test.pred.vector = apply(test.preds.matrix,1,sum)
  • $\begingroup$ Thanks for the comment! The other issue that I ran into while checking the k-fold cross validation code is that my model is a generalized linear mixed model containing 3 random effects and my data is in a hierarchical structure (GPS locations nested within individuals). I cannot seem to find R code to conduct a k-fold cross validation for a hierarchical data set that is analyzed under a GLMM. Any suggestions would be much appreciated! $\endgroup$
    – Buck2079
    Apr 6, 2015 at 1:00
  • $\begingroup$ try to include an example of your model code and data $\endgroup$ Apr 6, 2015 at 11:57
  • $\begingroup$ See example hierarchical data set and R code (GLMM model). I need to conduct a k-fold cross validation but need to use animals as a grouping variable to be withheld from the model validation procedure rather than pulling individiual locations from animals to validate the model's predictability. Does that make sense? $\endgroup$
    – Buck2079
    Apr 6, 2015 at 20:19
  • $\begingroup$ sry second part I'm not sure of $\endgroup$ Apr 6, 2015 at 20:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.