I have a binary classification problem where the fraction of positives is very low, e.g. 20 positives in 10,000 examples (0.2%)

What is an appropriate cross validation scheme for training a classifier with very few positives?

I currently have the following setup:

tmp <- createDataPartition(Y, p = 9/10, times = 3, list = TRUE)
myCtrl <- trainControl(method = "boot", index = tmp, timingSamps = 2, classProbs = TRUE, summaryFunction = twoClassSummary)

RFmodel <- train(X,Y,method='rf',trControl=myCtrl,tuneLength=1, metric="ROC")
SVMmodel <- train(X,Y,method='svmRadial',trControl=myCtrl,tuneLength=3, metric="ROC")
KNNmodel <- train(X,Y,method='knn',trControl=myCtrl,tuneLength=10, metric="ROC")
NNmodel <- train(X,Y,method='nnet',trControl=myCtrl,tuneLength=3, trace = FALSE, metric="ROC")

but I am not getting good performance (my ROC values are < 0.7 for all the classifiers above)

  • 1
    $\begingroup$ ROC curves < 0.7 may be just as good as you can possibly get with just 0.2% positives. The base rate matters a lot in classification: en.wikipedia.org/wiki/Base_rate_fallacy Nevertheless, +1 for an interesting question. $\endgroup$ – Stephan Kolassa Feb 11 '13 at 8:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.