2
$\begingroup$

I am running repeat simulations of leave-one-out cross-validation on glmnet models of randomly generated data, and collecting the AUC on left-out predictions (vs the full set of random targets). The data is randomly generated and uncorrelated.

This entire process is simulated 100 times, generating new data each time, to do an n-fold validation of glmnet modelling each time, all to collect a distribution of AUCs, one for each of the 100 simulations.

Code below.

"Why the heck is he doing this?" I hear you ask. I am trying to use this as a baseline for accuracy measures of models on real data with a small number of samples.

but, a very odd thing occurs :

  1. When the number of records is 30, about one in 10 cases can have an AUC of ZERO - ie, perfectly bad prediction (on left-out/out-of-sample cases !). This gets more pronounced for smaller numbers of records and fields.

  2. The AUC distribution is highly bimodal with a spike of zero and tiny AUCs. The bimodality diminishes with increasing records and columns.

  3. The median AUC is always well below 0.5 (which I would naively expect) while the maximum AUC is never above 0.75 - there is a directional bias to the AUC, surprising on cross-validated random data. This persists even as the zero AUCs stop, as they do when the number of records is 60.

What is going on here ? Is there a bug in my code, or is this some exotic pathology of cross-validation?

library(AUC)

nrecords <- 30 
nfeatures <- 2 
AUCvec <- NULL
for(j in 1:100)
    {

      RandomX <- matrix(rpois(nrecords*nfeatures,lambda=2000),nrow=nrecords)

      RandomY <- ifelse(rnorm(nrecords)>0,1,0)

      YY <- as.factor(RandomY)

      dat <- cbind(data.frame(RandomX),YY)

     cvpred <- NULL

     for(i in 1:nrow(dat)){

         cvg <- glm(YY ~ ., data=dat[-i,],family="binomial")

         cvpred[i] <- predict(cvg,dat[i,1:(nfeatures)])

     }

     AUCvec[j] <- auc(roc(cvpred,dat[,nfeatures+1]))

     print(c(AUCvec[j],length(which(RandomY==0))))



     }
hist(AUCvec)

table(AUCvec)

quantile(AUCvec,(0:10)/10)

added: Thanks to @khakieconomist Jim for editing my poorly formatted code, retweeting this post, and suggesting that I make it in the first place, as well as taking the time to confirm that the effect is real.

Thanks to Jim's retweet a paper was suggested by @kjhealy Kieran - it discusses issues of bias and variance with LOO-CV on AUC calculations and recommends using Leave Pair Out as a better solution, albeit a much more computationally complex one. I will certainly try this.

https://biblio.ugent.be/publication/860678/file/860679

Still nagging however is the fact that ZERO AUC can come up far more often than chance might suggest.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.