I'm training a binary classifier for disease detection.
Because of my small amount of data (~1000 datapoints, 10% positive, 90% negative), I've realized that doing an 80-20 train-test split produces quite a lot of variance in my results simply based off the test set I choose.
So I want to do k-fold cross validation. The problem is, I was doing threshold tuning (based off the ROC curve) when I was evaluating models before.
But now I'm not sure how to do threshold tuning - getting a new decision threshold for each fold feels weird - when I've tried this I've gotten a pretty big spread of thresholds. Plus what would my threshold be for my final model if doing this?
The problem is that the model trained for each fold seems to be calibrated differently, so when I tried taking the outputted probabilities for all the samples, and then choosing a threshold based off that, it didn't do well.
How can I decide on a decision threshold? Do I treat it like a hyperparameter, and identify the best one using k-fold?
Perhaps I could just work with the probabilities - but the problem is I'm using algorithms that don't output well-calibrated probabilities (neural nets, support vector machines) - and everything I've read says that I need a holdout set to calibrate probabilities, thus facing the same problem as for a decision threshold.
Any help or advice would be appreciated!