1
$\begingroup$

I'm training a binary classifier for disease detection.

Because of my small amount of data (~1000 datapoints, 10% positive, 90% negative), I've realized that doing an 80-20 train-test split produces quite a lot of variance in my results simply based off the test set I choose.

So I want to do k-fold cross validation. The problem is, I was doing threshold tuning (based off the ROC curve) when I was evaluating models before.

But now I'm not sure how to do threshold tuning - getting a new decision threshold for each fold feels weird - when I've tried this I've gotten a pretty big spread of thresholds. Plus what would my threshold be for my final model if doing this?

The problem is that the model trained for each fold seems to be calibrated differently, so when I tried taking the outputted probabilities for all the samples, and then choosing a threshold based off that, it didn't do well.

How can I decide on a decision threshold? Do I treat it like a hyperparameter, and identify the best one using k-fold?

Perhaps I could just work with the probabilities - but the problem is I'm using algorithms that don't output well-calibrated probabilities (neural nets, support vector machines) - and everything I've read says that I need a holdout set to calibrate probabilities, thus facing the same problem as for a decision threshold.

Any help or advice would be appreciated!

Improve this question
$\endgroup$
3
  • $\begingroup$ One first idea, use stratified sampling when doing CV. $\endgroup$
    – user2974951
    Dec 16, 2022 at 13:52
  • $\begingroup$ I would indeed recommend that you stick with the probabilistic classifications. Yes, you should use a holdout sample to calibrate them - but you can simulate this by doing cross-validation, just as for any other model training task. Assess them using proper scoring rules. Find "optimal" thresholds for your final model, using cost-based trade-offs. $\endgroup$
    – Stephan Kolassa
    Dec 16, 2022 at 16:33
  • $\begingroup$ Hi Stephan - thanks so much for your comment. Can you elaborate a bit? How do I find a threshold for my final model, if it's been trained on all my data (so I don't have a holdout set)? $\endgroup$
    – jimbo
    Dec 16, 2022 at 19:01

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.