The title explains my problem. I cannot calculate AUC as things currently stand because some of my data sets have only negatives. I have three possible solutions and I need input from more experienced statisticians. I am still a learner and my research has not revealed to me an appropriate methodology to address this issue.
I am doing
k fold cross validation in R where
k = 1. I have 12 independent data sets (which recorded the same observations). Therefore, I will have a fold that looks like this which will give me problems:
I've got some R code to calculate the AUC. Given the above, I use the training for my
testingDataSet is comprised of data set 4.
model<-glm(a ~ b + c + d, family = binomial(link = 'logit'), data = trainingDataSet) p <- predict(model, newdata=testingDataSet, type="response") pr <- prediction(p, testingDataSet$a)
The error it gives is as follows:
Error in prediction(p, testingDataSet$a) : Number of classes is not equal to 2. ROCR currently supports only evaluation of binary classification tasks.
This is because my testingData only has negative numbers. I'll provide the following R code to show you. Please see:
> unique(testingDataSet$a)  0
Perhaps a more robust way to show you is to show you the entire training set variable we are trying to predict for:
> testingDataSet$a  0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
Thus, we have a training set that is comprised of all negatives. I have another fold that is similar.
I have three solutions that I believe can solve the issues I am facing. Clearly, the last two proposed solutions I think are very poor solutions, but some I have read in data science forums and included them here to be thorough. That said, I feel they are very ill advised.
Throw out k cross validation for
k = 1and
k = 2. I have two datasets that have all negatives. I need testing and training validations that have both negatives and positives. I can then choose
k = 3and
k = 4for instance.
Remove the datasets that have all negative results. (Bad idea)
Compute AUC as 5.0 when given a data set with all negative results. (Bad idea)
This is the first time I have encountered this problem. I deeply appreciate your mentorship and assistance to a learner and what is considered best practice in this scenario. I think I have found it above, but definitely open to further suggestions or more formal statistical methodology here that I am not aware of. Thank you for your time and for reading this question.