0
$\begingroup$

I have a binary classification model which returns class probability scores as the outcome. I did the model parameter tuning using repeated k-fold cross-validation. I am wondering how can I decide the optimal cutoff probability for my final model. Take the Youden index for example, can I use the average cutoff threshold determined by Youden index for each cross-validation fold?

Thanks!

$\endgroup$
  • $\begingroup$ Much depends here on what you are trying to "optimize" with your "optimal cutoff probability." If you need a cutoff, you need to specify for example what the relative costs of false-negative and false-positive classifications are. Youden's index "gives equal weight to false positive and false negative values." That might not be appropriate for your application. Please edit your question to say more about what you are trying to accomplish with your model and how it might be used in practice. $\endgroup$ – EdM Nov 7 '18 at 21:17
  • $\begingroup$ Yes, you are right, it muchly depends on how we define 'optimal'. Here, what I am more interested in is for a given optimality metric (e.g. Youden index), how can I determine the cutoff threshold from cross-validation? $\endgroup$ – Doodle Nov 7 '18 at 21:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.