Are there better estimators of misclassification error than the fraction of misclassified test points?

Question

Assume we train a binary classification model using the training set. Also assume that the model returns an estimate of the probability of success $\hat f(x)$ for every feature vector $x$ and was trained with "an intent of" minimizing out of sample cross-entropy error (maximizing likelihood). Moreover, assume we actually picked the algorithm based on the training data (but never looked at the test data), so we don't trust cross-validation (because training data can't be "unseen" by researcher who created an algorithm to train $\hat f$).

We are interested in the estimation of the out of sample misclassification error $E_{out} = P((\hat f(x) \geq 0.5) \neq y)$. The commonly used approach is to interpret the number of misclassified points $n_{\text{test,misclassified}}$ in the test set as an observation of a binomial random variable with probability $p = E_{out}$ and number of trials $n = n_{\text{test}}$ (number of points in the test set). Then the classical estimate would be $\hat E_{out} = n_{\text{test,misclassified}} / n_{\text{test}}$. This gives an estimator with 0 bias but potentially high variance (if the test set is small). We may want to use additional knowlede we have to reduce the variance. We have e.g.:

1. Test and training sets,
2. Estimated probabilities $\hat f(x)$.

Are there any commonly occurring situations where we can leverage any additional knowledge to provide a better estimate of misclassification error?

Answer 1

If you are really worried about unaccounted "researcher degrees of freedom", I suspect the best you can do is to automate your model construction procedure (including feature selection, hyper-parameter optimisation etc.), so that the researcher is not making any choices after seeing the data. You can then use cross-validation provided you perform the whole model construction procedure independently in each fold.

In practice, it is often best to simply acknowledge there are unaccounted degrees of freedom, and make an effort not to over-analyse the data. While AutoML has become a hot topic in machine learning, I suspect "CyborgML", where the operator becomes an implicit and unaccounted part of the machine learning procedure, is likely to be a significant issue for the foreseeable future, especially in Deep Learning, where models are computationally expensive to fit. This ought to be less of a problem for more classical statistics.

Answer 2

$Eout=P((\hat f(x)≥0.5)≠y) $ is rarely used. You're assuming that 0.5 is a useful decision threshold. What if all of $\hat f(x)$ is above 0.5, or all below?

Because $\hat f(x)$ is continuous, you can set this threshold in many different ways. At one, true positive rate is maximized. At the other, false positive rate is maximized. Somewhere in between F1 is maximized. Which is best, really depends on your application.

For classification problems, AUC and Average Precision scores ingrate over all decision thresholds to give you a single number.

However these have their own problems (as do TPR and FPR). AUC is a terrible metric when class population size are imbalanced.

But to answer your specific question, which I understood as, how can I evaluate my algorithm without being 'polluted' by train data. Cross validation is the gold standard here. You separate the train/test into N non-overlapping pairs of train and test. The trainer never see the test data. You end up with k models and k independent tests.

In the extreme case, you can set the test set size=1, and train on the rest. If you assume this tiny difference in training data has little effect on the model, your test set size can be as large as your training set size (although small test sets aren't suitable for calculating AUC).

Confidence intervals on TPR, FPR, AUC can also be calculated and are useful. See: binomial CI and AUC CI