2
$\begingroup$

I recently ran a k-fold cross validation on a data set/model that I was interested in evaluating the performance of. In doing so, I received a value of 0.46. I'm assuming this low value indicates poor model predictability but wasn't 100% sure. Thanks for the help!


enter image description here

results_full <- glmer(R0A1~MP_Scaled+MPHW_Scaled+HW_Scaled+YP_Scaled+AG_Scaled+Shrub_Scaled
    +(1|ID)+(1|Site)+(Year),
    data=secondorder, family=binomial)
summary(results_full)
$\endgroup$
  • $\begingroup$ 0.46 what? What is your metric? Accuracy? $\endgroup$ – Marc Claesen Feb 6 '15 at 16:44
  • $\begingroup$ 46% correct classification $\endgroup$ – Buck2079 Feb 6 '15 at 17:26
  • $\begingroup$ @Buck2079 Did you find a way to perform k-fold cross validation of GLMM's in R? I am having the same problem now... Thank you in advance!! $\endgroup$ – mtao Jul 13 '16 at 10:43
2
$\begingroup$

It completely depends on the situation. You have to understand the context of your data or at least know how well others have performed before you. For balanced binary classification, then 46% is worse than tossing a coin. But let's say your model have identified a small group of start-up companies, where 46% increased their assets more than 10-fold the following year. Then the cost of false-positive is negligible and your model is performing incredibly well.

simple k-fold cross-validation code:

library(foreach)
obs=200
vars=80
kfold=5
noise.factor = 2

#a data.matrix
X = replicate(vars,rnorm(obs))
true.coefs = rnorm(vars)
y = as.numeric(true.coefs %*% t(X)) + rnorm(obs) * noise.factor

#a simple kfold code
folds = split(sample(1:length(y)),1:kfold)
test.preds.matrix = foreach(i = folds,.combine=cbind) %do% {
 Data.train = data.frame(X=X[-i,],Y=y[-i])
 Data.test  = data.frame(X=X[i ,],Y=y[ i])
 lmf = lm(Y~.,Data.train)  
 test.pred = rep(0,length(y))
 test.pred[i] = predict(lmf,Data.test)
 return(test.pred)
}
#diagnostics plot
test.pred.vector = apply(test.preds.matrix,1,sum)
plot(test.pred.vector,y,main=paste("R²=",round(cor(test.pred.vector,y),2)))
$\endgroup$
  • $\begingroup$ Thanks for the comment! The other issue that I ran into while checking the k-fold cross validation code is that my model is a generalized linear mixed model containing 3 random effects and my data is in a hierarchical structure (GPS locations nested within individuals). I cannot seem to find R code to conduct a k-fold cross validation for a hierarchical data set that is analyzed under a GLMM. Any suggestions would be much appreciated! $\endgroup$ – Buck2079 Apr 6 '15 at 1:00
  • $\begingroup$ try to include an example of your model code and data $\endgroup$ – Soren Havelund Welling Apr 6 '15 at 11:57
  • $\begingroup$ See example hierarchical data set and R code (GLMM model). I need to conduct a k-fold cross validation but need to use animals as a grouping variable to be withheld from the model validation procedure rather than pulling individiual locations from animals to validate the model's predictability. Does that make sense? $\endgroup$ – Buck2079 Apr 6 '15 at 20:19
  • $\begingroup$ sry second part I'm not sure of $\endgroup$ – Soren Havelund Welling Apr 6 '15 at 20:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.