How to evaluate a classifier on a very small dataset? I have a very small dataset (18 (pos) + 46 (neg) = 64) and a binary classification problem. I managed to build a classifier, but how should I assess the performance of that model? Specifically, I'd like to measure Sensitivity and Specificity (recalls) and corresponding confidence intervals. I come up with several different options and now can't choose the appropriate:

*

*Measure the performance on a test set. This implies too high variance since there will be about 5-7 samples of the pos- class in the test set; also no CI.

*Cross-validation (3/4/5-folds). Seem to be better but the variance is still too high, and CI will be too wide.

*Leave-one-out cross-validation. It looks like a solution, also offered here. What about CI? Can I use the standard p(1-p)/sqrt(n) formula here (where p = precision/recall/... for the pos-/neg- class and n = number of samples in pos-/neg- class)?

*Use some data-generation techniques like SMOTE to generate new data points. However, we use it to balance the training set, and evaluation on such points leads to overestimating the true predictive power. Am I right?

*Make a train/test split, say, 1000 times, and calculate 1000 metrics. So we'll reduce variance by increasing sqrt(n) in the denominator of standard error.

*Use Monte-Carlo simulation to generate new data points with the same distributions and the same covariance matrices. I've been offered to use this method, but I'm not sure about that -  Datasaurus has (almost) the same covariance matrix on every dataset, but classification problems look completely different.

So, when should I prefer one to another? (for extra small datasets) Different answers here on CV offer different solutions.
I hope this question wouldn't be considered too broad, and thanks in advance!
 A: IMHO your best bet is repeated k-fold cross validaton or out-of-bootstrap (possibly also .632+-bootstrap, depending on the actual risk of overfitting).

*

*These resampling methods test in turn all cases, and that's the best in terms of CI you can get with such a small data set.

*They also test an arbitrarily large number of surrogate models, allowing you to measure model instability, and due to the large number of surrogate models that are pooled into the final result, reduce this component of the total variance uncertainty.



*

*Your CI will be dominated by the variance due to having at the very best only 18 (sensitivity) and 46 (specificity) real, independent cases in the respective tests.


*The binomial (e.g. p(1-p)/sqrt(n)) confidence intervals account for this variance, but the normal approximation you refer to is not appropriate with that few cases in the denominator of the figure of merit. There are better approximations, e.g. the Agresti-Coull method. In case you work in R, package binom offers a wide variety of such methods.


*You are right, a single train/test split will lead to so uncertain test results that you won't be able to draw any meaningful conclusions - the CI will then be e.g. "somewhere between guessing and perfect".


*LOO vs. k-fold cross validation will likely not matter in terms of confidence interval width: the number of tested cases is always n.
With k-fold CV, in particular repeated k-fold CV, however, you can in addition check model stability (another source of variance uncertainty). If it is not negligible compared to the binomial variance, you can also reduce it by doing more repetitions.
Repetitions of cross validation (or bootstrap), however, will not reduce that binomial variance since it is due to the finite number of independent tested cases (after you have sufficient surrogate models so that each case has been tested).
Thus, your point 5 is largely wrong here: the n that refers to the variance due to sample size will not increase above 18 or 46, the n that does increase refers to the number of surrogate models tested, i.e. model instability.


*Data generation, whether approach 4 or 6 is risky. In particular if your knowledge about the data is based on 18 and 46 cases only. Yes, expect unknown but possibly heavy optimistic bias.
IMHO you may do that, but the results should be reported separately from cross validation/out-of-bootstrap tests.
I may add that I see far more of a point in simulating known artifacts that can happen with this kind of data (perturb real data according to known error/failure mechanisms) in order to check ruggedness of the predictions.
A: Since you have a small dataset I would go for the leave-one-out.
To estimate the CI of the predictive performance you can perform a bootstrapping on the predictions.
(EDIT: the explanation of the bootstrap procedure was not clear)
For $N$ repetitions, you sample from your dataset with replicates and calculate the performance metrics using LOO. In the end, you have $N$ metrics from which you can calculate your CI.
You may want to have a look at this paper
Annette M. Molinaro, Richard Simon, Ruth M. Pfeiffer, Prediction error estimation: a comparison of resampling methods, Bioinformatics, Volume 21, Issue 15, Pages 3301–3307, https://doi.org/10.1093/bioinformatics/bti499
They show that LOO is the method that performs better in the tested models.

The .632+ bootstrap is quite biased in small sample sizes with strong signal-to-noise ratios

An R function to perform the LOO-Boot:
https://rdrr.io/cran/sortinghat/man/errorest_loo_boot.html
