When is an AUC score misleadingly high? I have an algorithm which gives an AUC (area under the receiver operating curve) of 0.94.
I mean, this is amazing, but... probably too amazing, considering the difficulty of the task I am working on. So how can I tell if the AUC is valid, or misleadingly high?
(P.S. yes, I am training on the training set and testing on the completely separate testing set.)
 A: One possible reason you can get high AUROC with what some might consider a mediocre prediction is if you have imbalanced data (in favor of the "zero" prediction), high recall, and low precision. That is, you're predicting most of the ones at the higher end of your prediction probabilities, but most of the outcomes at the higher end of your prediction probabilities are still zero. This is because the ROC score still gets most of its "lift" at the early part of the plot, i.e., for only a small fraction of the zero-predictions. 
For example, if 5% of the test set are "ones" and all of the ones appear in the top 10% of your predictions, then your AUC will be at least 18/19 because, after 18/19 of the zeroes are predicted, already 100% of the ones were predicted. Even if the top 5% are all zeroes.
A simple python example:
import sklearn
import numpy as np

yTest = [0,0,1,1,0,1,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0]
yPredicted = np.linspace(0.9, 0.1, num=len(yTest))
sklearn.metrics.roc_auc_score(yTest, yPredicted) # ~0.89

import matplotlib.pyplot as plt
fpr, tpr, threshold = sklearn.metrics.roc_curve(yTest, yPredicted)
plt.plot(fpr, tpr)


Whether this is a "bad" prediction depends on your priorities. If you think that false negatives are terrible and false positives are tolerable, then this prediction is okay. But if it's the opposite, then this prediction is pretty bad.
A: Datasets used for predictive modeling rarely have the variability necessary to develop and evaluate a model as widely applicable as others believe it to be. There are encroaching expectations that models be applicable across seasons, years, in different settings, and among different populations. It doesn't matter whether one uses bootstrapping, cross-validation, or split sample validation when the dataset itself is neither independent nor representative of its target application. 
Just the other day, I looked at a patent for an ML algorithm that promised to classify cancerous nodes from radiographic imaging of the lungs. But the cancer cases that were sampled came from a largely white, male, older population... and there were no controls (non-cancer screens). The disconnect between development and application is appalling, there's a complete lack of scientific thought.
A: Agree with Dan, it could be that your dataset on has a 6% event rate, so 94% 0 and 6% 1's, so the dataset is imbalanced. The model without weights and with a cut-off value of 0.5, will come back as everything predicted as 0 and so will have ~94% accuracy. 
If you try using a suitable cut-off value and then recompute the confusion matrix and other metrics, you should get a more resonable result.
