I have a binary classification problem where I perform cross validation on the training set (currently 80% of the examples) and then evaluate results on a test set.
I use cross validation for finding the best algorithm and its optimal parameters (I use AUC score as performance indicator of the folds), but then I need to choose a threshold between 0 and 1 to "complete" the classification pipeline.
QUESTION: which is the right classification pipeline step where I should choose the prediction threshold?
I have in mind 3 options:
- Choose the prediction threshold on the basis of the prediction probabilities f the test set: I've seen doing this but it doesn't look that correct, am I wrong?
- Create a validation set (so I could split the data with ratios 60/20/20 for training, validation, test sets) and choose the optimal threshold on it; then apply this threshold, together with the best algorithm found with cross validation, on the test set. The issue with this solution is that my data is not that much...
- Consider the threshold prediction as one of the parameters to be chosen during the cross validation. Since a threshold grid search on my data would be pretty expensive, I was thinking to perform the CV in 2 steps:
- a first cross validation to select the model and its parameters
- then, given the best model/parameters, perform a second cross validation to choose the threshold, by using f1 score as performance metric
Which, among the above options, looks feasible? Which is the best/correct approach?