After training a classification model, we rarely use the default 0.5 probability cutoff in real use cases. Often the optimal cutoff is determined by expected value analysis. If the cost of false positive and false negative is unknown, sometime I use the cutoff that gives the highest Youden Index (sensitivity+specificity-1).
When data is split into training, validation and test set, after the model trained on training set, the cutoff is determined on validation set. When there are not that many data samples and cannot afford to split a dedicated validation set, I do cross-validation. But in this case, what is the correct way to determine the cutoff? In cross-validation, there are multiple 'validation set'(folds).
What I did in my last project was that I concatenated all the predicted score from all folds, and determine the cutoff using these 'cv predictions', in my case using the cutoff with highest Youden Index. I wonder if this is the right way to do?
I appreciate it if anyone can clarify me on this. Thanks a lot.