Assume we train a binary classification model using the training set. Also assume that the model returns an estimate of the probability of success $\hat f(x)$ for every feature vector $x$ and was trained with "an intent of" minimizing out of sample cross-entropy error (maximizing likelihood). Moreover, assume we actually picked the algorithm based on the training data (but never looked at the test data), so we don't trust cross-validation (because training data can't be "unseen" by researcher who created an algorithm to train $\hat f$).

We are interested in the estimation of the out of sample misclassification error $E_{out} = P((\hat f(x) \geq 0.5) \neq y)$. The commonly used approach is to interpret the number of misclassified points $n_{\text{test,misclassified}}$ in the test set as an observation of a binomial random variable with probability $p = E_{out}$ and number of trials $n = n_{\text{test}}$ (number of points in the test set). Then the classical estimate would be $\hat E_{out} = n_{\text{test,misclassified}} / n_{\text{test}}$. This gives an estimator with 0 bias but potentially high variance (if the test set is small). We may want to use additional knowlede we have to reduce the variance. We have e.g.:

  1. Test and training sets,
  2. Estimated probabilities $\hat f(x)$.

Are there any commonly occurring situations where we can leverage any additional knowledge to provide a better estimate of misclassification error?

  • $\begingroup$ Not trusting cross validation would seem to imply you have larger issues. $\endgroup$ – Matthew Drury Jul 14 '17 at 17:30
  • $\begingroup$ Why? Cross validation results are only fully trust-able when the training process can isolate arbitrary subset of the data as a test set, which is not the case when the model choice is performed by a researcher looking at the data. No? $\endgroup$ – fiktor Jul 15 '17 at 22:01
  • $\begingroup$ Why should the researcher's choice of model affect the test set? Sepearate your train/test set with a random number generator. $\endgroup$ – user48956 Jul 17 '17 at 19:52
  • $\begingroup$ I'm saying that the test set should best be picked before researcher does exploratory data analysis. Otherwise rigourously speaking test set error can no longer be trusted since the model choice is not independent of the test set. $\endgroup$ – fiktor Jul 17 '17 at 23:45

$Eout=P((\hat f(x)≥0.5)≠y) $ is rarely used. You're assuming that 0.5 is a useful decision threshold. What if all of $\hat f(x)$ is above 0.5, or all below?

Because $\hat f(x)$ is continuous, you can set this threshold in many different ways. At one, true positive rate is maximized. At the other, false positive rate is maximized. Somewhere in between F1 is maximized. Which is best, really depends on your application.

For classification problems, AUC and Average Precision scores ingrate over all decision thresholds to give you a single number.

However these have their own problems (as do TPR and FPR). AUC is a terrible metric when class population size are imbalanced.

But to answer your specific question, which I understood as, how can I evaluate my algorithm without being 'polluted' by train data. Cross validation is the gold standard here. You separate the train/test into N non-overlapping pairs of train and test. The trainer never see the test data. You end up with k models and k independent tests.

In the extreme case, you can set the test set size=1, and train on the rest. If you assume this tiny difference in training data has little effect on the model, your test set size can be as large as your training set size (although small test sets aren't suitable for calculating AUC).

Confidence intervals on TPR, FPR, AUC can also be calculated and are useful. See: binomial CI and AUC CI

| cite | improve this answer | |
  • $\begingroup$ Why is AUC terrible when class population is unbalanced? It is the probability the model scores a random positive class higher than a random negative class, it seems to me it is balance agnostic. $\endgroup$ – Matthew Drury Jul 14 '17 at 17:29
  • $\begingroup$ Well -- I have an Alzheimer model with an AUC of 0.87. Sounds respectable. Is it useful? In this population (which includes a lot of young people. The model derives this AUC mainly from looking at the age feature. Young people rarely get Alzheimer's. Is this a good test for Alzheimer disease? Not really -- here's a couple of sample decision cuts: accuracy=98.8%, tpr=22.9% fpr=1% ppv=2% npv=99.8%. At the other end: accuracy=30%, tpr=98.% fpr=70% ppv=0.01% npv=99.9%. The AUC is reasonable, but age is not a good test for alheimer's $\endgroup$ – user48956 Jul 17 '17 at 19:30
  • $\begingroup$ ppv=2% basically means that if the model says you will get Alzheimer's, it will be right just 2% of the time. :-/ The average precision score for this same model is 0.05. This is a good reflection of the model's poor ppv, across many different FPRs. $\endgroup$ – user48956 Jul 17 '17 at 19:33
  • $\begingroup$ ncbi.nlm.nih.gov/pmc/articles/PMC4403252 -- argues for AP scores where the dataset is imbalanced. "The AUC lacks sensitivity in identifying cases". In my Alzheimer case, the model gets high AUC by (correctly) identifying young people won't get Alzheimer's. But performance for older people, where all the positive cases are, its basically, a crap shoot. $\endgroup$ – user48956 Jul 17 '17 at 19:45
  • 1
    $\begingroup$ If you're goal is to identify alzheimers in the older population, why not just leave the young out of the training and testing sets? $\endgroup$ – Matthew Drury Jul 17 '17 at 20:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.