K-fold cross validation

Question

I recently ran a k-fold cross validation on a data set/model that I was interested in evaluating the performance of. In doing so, I received a value of 0.46. I'm assuming this low value indicates poor model predictability but wasn't 100% sure. Thanks for the help!

---

results_full <- glmer(R0A1~MP_Scaled+MPHW_Scaled+HW_Scaled+YP_Scaled+AG_Scaled+Shrub_Scaled
    +(1|ID)+(1|Site)+(Year),
    data=secondorder, family=binomial)
summary(results_full)

Answer 1

It completely depends on the situation. You have to understand the context of your data or at least know how well others have performed before you. For balanced binary classification, then 46% is worse than tossing a coin. But let's say your model have identified a small group of start-up companies, where 46% increased their assets more than 10-fold the following year. Then the cost of false-positive is negligible and your model is performing incredibly well.

simple k-fold cross-validation code:

library(foreach)
obs=200
vars=80
kfold=5
noise.factor = 2

#a data.matrix
X = replicate(vars,rnorm(obs))
true.coefs = rnorm(vars)
y = as.numeric(true.coefs %*% t(X)) + rnorm(obs) * noise.factor

#a simple kfold code
folds = split(sample(1:length(y)),1:kfold)
test.preds.matrix = foreach(i = folds,.combine=cbind) %do% {
 Data.train = data.frame(X=X[-i,],Y=y[-i])
 Data.test  = data.frame(X=X[i ,],Y=y[ i])
 lmf = lm(Y~.,Data.train)  
 test.pred = rep(0,length(y))
 test.pred[i] = predict(lmf,Data.test)
 return(test.pred)
}
#diagnostics plot
test.pred.vector = apply(test.preds.matrix,1,sum)
plot(test.pred.vector,y,main=paste("R²=",round(cor(test.pred.vector,y),2)))