I have all negatives in (some) of my folds in cross validation. Seeking best way to calculate AUC using R in this scenario?

Problem

The title explains my problem. I cannot calculate AUC as things currently stand because some of my data sets have only negatives. I have three possible solutions and I need input from more experienced statisticians. I am still a learner and my research has not revealed to me an appropriate methodology to address this issue.

Further Details

I am doing k fold cross validation in R where k = 1. I have 12 independent data sets (which recorded the same observations). Therefore, I will have a fold that looks like this which will give me problems:

training: 1,2,3,5,6,7,8,9,10,11,12 testing: 4

I've got some R code to calculate the AUC. Given the above, I use the training for my trainingDataSet. The testingDataSet is comprised of data set 4.

model<-glm(a ~ b + c + d, family = binomial(link = 'logit'), data = trainingDataSet)

p <- predict(model, newdata=testingDataSet, type="response")

pr <- prediction(p, testingDataSet$a) The error it gives is as follows: Error in prediction(p, testingDataSet$a) :
Number of classes is not equal to 2.
ROCR currently supports only evaluation of binary classification tasks.

This is because my testingData only has negative numbers. I'll provide the following R code to show you. Please see:

testingDataSet$a. > unique(testingDataSet$a)
 0

Perhaps a more robust way to show you is to show you the entire training set variable we are trying to predict for:

> testingDataSet\$a
 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

Thus, we have a training set that is comprised of all negatives. I have another fold that is similar.

Question

I have three solutions that I believe can solve the issues I am facing. Clearly, the last two proposed solutions I think are very poor solutions, but some I have read in data science forums and included them here to be thorough. That said, I feel they are very ill advised.

• Throw out k cross validation for k = 1 and k = 2. I have two datasets that have all negatives. I need testing and training validations that have both negatives and positives. I can then choose k = 3 and k = 4 for instance.

• Remove the datasets that have all negative results. (Bad idea)

• Compute AUC as 5.0 when given a data set with all negative results. (Bad idea)

This is the first time I have encountered this problem. I deeply appreciate your mentorship and assistance to a learner and what is considered best practice in this scenario. I think I have found it above, but definitely open to further suggestions or more formal statistical methodology here that I am not aware of. Thank you for your time and for reading this question.

A couple things...

AUC (generally) provides a measure of prediction accuracy when comparing a continuous prediction (say, probability between 0 and 1) to a binary response (true 0 or 1), which I assume is what you have going on here. The closer to 1 and 0 that your model predicts for true 1s and 0s, respectively, the higher the AUC. A random model results in AUC = 0.5 (not 5.0).

Generally we would refer to "k" as the number of folds in your validation (so you can't really validate with k=1). You actually seem to have k=14, and you're witholding 1 fold at a time (which is typical).

More information on your actual data would be useful. As is, I can only speak in generalities.

The problem...

AUC works by comparing (at a full breadth of thresholds) the false positive (FP) rate with the true positive (TP) rate. If your threshold is 0, all predictions will = 1, all true 1s will be TPs, but all true 0s will be FPs. With a threshold of 1, all predictions will = 0, and the opposite outcome will occur.

In order to calculate this balance of TP and FP, AUC requires at least one of each true binary value (0 or 1), otherwise it has no data on one of the two: FP or TP. So you can't run the test with only a single unique value of truth. So you need to figure out a different way of defining your folds. But how do you do that?

Without any decent information on exactly what you're measuring, it's impossible to say for sure. One thing is certain: you need the folds to be independent. In other words, you can't just randomly assign your data into folds and expect an unbiased validation. Issues of data structure and autocorrelation will make this validation overly-optimistic (you'll think your model is better than it is). You call your folds "independent", but if you have underlying structure in your predictors, you model may still be overfit, which will also bias your validations.

The solution...

One simple strategy would be to group folds together, ensuring that you have at least one (preferably more) of each true response (0 and 1) in each fold. The closer to equal prevalence you have (equal 0s and 1s), the more reliable the AUC will be (I believe, but check on that).

There is no hard-and-fast rule about how many folds you should use (i.e. the ideal value of k). Hastie et al (2009) and Kohavi (1995) have discussed this. Ideally, you want folds 1) to be [as] independent [as possible], 2) to contain enough data to make reasonable validations, and 3) to not force the model to extrapolate (i.e. in predictor space) when predicting to the withheld data (unless you specifically want to test this).

When grouping folds together, to maintain as much independence as possible, you'll want the most similar folds to be grouped together. This will maximise the independence between the training and testing data (see Roberts et al 2016). By this, I don't mean the most similar data values. I mean the most similar in quality (e.g. similar location, same experimental units, similar time, etc.). It's impossible to say specifically, as you've given no information on your data.

Without a better understanding of your actual analysis, that's about all the help I can offer.

Hastie, T. et al. 2009. "The Elements of Statistical Learning: Data Mining, Inference, and Prediction." Springer Verlag.

Kohavi, R. 1995. "A study of cross-validation and bootstrap for accuracy estimation and model selection." The Fourteenth International Joint Conference on Artifical Intelligence. Morgan Kaufmann Publishers Inc. San Francisco, CA, USA, pp. 1137-1143.

Roberts, D. R., V. Bahn, S. Ciuti, M. S. Boyce, J. Elith, G. Guillera-Arroita, S. Hauenstein, J. J. Lahoz-Monfort, B. Schröder, W. Thuiller, D. I. Warton, B. A. Wintle, F. Hartig, and C. F. Dormann. 2017 "Cross-validation strategies for data with temporal, spatial, hierarchical, or phylogenetic structure." Ecography, in press.