I have a very small dataset (18 (pos) + 46 (neg) = 64) and a binary classification problem. I managed to build a classifier, but how should I assess the performance of that model? Specifically, I'd like to measure Sensitivity and Specificity (recalls) and corresponding confidence intervals. I come up with several different options and now can't choose the appropriate:
- Measure the performance on a test set. This implies too high variance since there will be about 5-7 samples of the pos- class in the test set; also no CI.
- Cross-validation (3/4/5-folds). Seem to be better but the variance is still too high, and CI will be too wide.
- Leave-one-out cross-validation. It looks like a solution, also offered here. What about CI? Can I use the standard
p(1-p)/sqrt(n)formula here (where p = precision/recall/... for the pos-/neg- class and n = number of samples in pos-/neg- class)?
- Use some data-generation techniques like SMOTE to generate new data points. However, we use it to balance the training set, and evaluation on such points leads to overestimating the true predictive power. Am I right?
- Make a train/test split, say, 1000 times, and calculate 1000 metrics. So we'll reduce variance by increasing sqrt(n) in the denominator of standard error.
- Use Monte-Carlo simulation to generate new data points with the same distributions and the same covariance matrices. I've been offered to use this method, but I'm not sure about that - Datasaurus has (almost) the same covariance matrix on every dataset, but classification problems look completely different.
So, when should I prefer one to another? (for extra small datasets) Different answers here on CV offer different solutions.
I hope this question wouldn't be considered too broad, and thanks in advance!