7
$\begingroup$

I have a binary classification problem where I perform cross validation on the training set (currently 80% of the examples) and then evaluate results on a test set.

I use cross validation for finding the best algorithm and its optimal parameters (I use AUC score as performance indicator of the folds), but then I need to choose a threshold between 0 and 1 to "complete" the classification pipeline.

QUESTION: which is the right classification pipeline step where I should choose the prediction threshold?

I have in mind 3 options:

  1. Choose the prediction threshold on the basis of the prediction probabilities f the test set: I've seen doing this but it doesn't look that correct, am I wrong?
  2. Create a validation set (so I could split the data with ratios 60/20/20 for training, validation, test sets) and choose the optimal threshold on it; then apply this threshold, together with the best algorithm found with cross validation, on the test set. The issue with this solution is that my data is not that much...
  3. Consider the threshold prediction as one of the parameters to be chosen during the cross validation. Since a threshold grid search on my data would be pretty expensive, I was thinking to perform the CV in 2 steps:
    • a first cross validation to select the model and its parameters
    • then, given the best model/parameters, perform a second cross validation to choose the threshold, by using f1 score as performance metric

Which, among the above options, looks feasible? Which is the best/correct approach?

Thanks!!

Improve this question
$\endgroup$
3
  • $\begingroup$ Recommended reading: stats.stackexchange.com/questions/464636/… (Disclosure: I posted that question.) $\endgroup$
    – Dave
    May 30, 2020 at 20:16
  • 1
    $\begingroup$ thank you! very interesting discussion. nevertheless, I can't see there an answer to my original question: am I wrong? $\endgroup$
    – user1403546
    May 31, 2020 at 14:53
  • 1
    $\begingroup$ It doesn’t answer your question but is important to read and understand before you make hard decisions. $\endgroup$
    – Dave
    May 31, 2020 at 16:30

1 Answer 1

Reset to default
2
$\begingroup$

Go with 3:

  • wrt 1 you are correct - this makes the test set part of the training of the actual classifier
  • 2 is a waste of cases that doesn't gain you anything over 3
Improve this answer
$\endgroup$
1
  • $\begingroup$ thank you! it corresponds to my initial hypothesis $\endgroup$
    – user1403546
    Jun 1, 2020 at 8:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.