K-fold cross validation

I recently ran a k-fold cross validation on a data set/model that I was interested in evaluating the performance of. In doing so, I received a value of 0.46. I'm assuming this low value indicates poor model predictability but wasn't 100% sure. Thanks for the help!

results_full <- glmer(R0A1~MP_Scaled+MPHW_Scaled+HW_Scaled+YP_Scaled+AG_Scaled+Shrub_Scaled
+(1|ID)+(1|Site)+(Year),
data=secondorder, family=binomial)
summary(results_full)

• 0.46 what? What is your metric? Accuracy? Feb 6, 2015 at 16:44
• 46% correct classification Feb 6, 2015 at 17:26
• @Buck2079 Did you find a way to perform k-fold cross validation of GLMM's in R? I am having the same problem now... Thank you in advance!! Jul 13, 2016 at 10:43

It completely depends on the situation. You have to understand the context of your data or at least know how well others have performed before you. For balanced binary classification, then 46% is worse than tossing a coin. But let's say your model have identified a small group of start-up companies, where 46% increased their assets more than 10-fold the following year. Then the cost of false-positive is negligible and your model is performing incredibly well.

simple k-fold cross-validation code:

library(foreach)
obs=200
vars=80
kfold=5
noise.factor = 2

#a data.matrix
X = replicate(vars,rnorm(obs))
true.coefs = rnorm(vars)
y = as.numeric(true.coefs %*% t(X)) + rnorm(obs) * noise.factor

#a simple kfold code
folds = split(sample(1:length(y)),1:kfold)
test.preds.matrix = foreach(i = folds,.combine=cbind) %do% {
Data.train = data.frame(X=X[-i,],Y=y[-i])
Data.test  = data.frame(X=X[i ,],Y=y[ i])
lmf = lm(Y~.,Data.train)
test.pred = rep(0,length(y))
test.pred[i] = predict(lmf,Data.test)
return(test.pred)
}
#diagnostics plot
test.pred.vector = apply(test.preds.matrix,1,sum)
plot(test.pred.vector,y,main=paste("R²=",round(cor(test.pred.vector,y),2)))

• Thanks for the comment! The other issue that I ran into while checking the k-fold cross validation code is that my model is a generalized linear mixed model containing 3 random effects and my data is in a hierarchical structure (GPS locations nested within individuals). I cannot seem to find R code to conduct a k-fold cross validation for a hierarchical data set that is analyzed under a GLMM. Any suggestions would be much appreciated! Apr 6, 2015 at 1:00
• try to include an example of your model code and data Apr 6, 2015 at 11:57
• See example hierarchical data set and R code (GLMM model). I need to conduct a k-fold cross validation but need to use animals as a grouping variable to be withheld from the model validation procedure rather than pulling individiual locations from animals to validate the model's predictability. Does that make sense? Apr 6, 2015 at 20:19
• sry second part I'm not sure of Apr 6, 2015 at 20:41