2
$\begingroup$

Assume I have a dataset split into train/val/test and I want to compute the optimal threshold value for an F1 score. This threshold value is in [0, 0.5] as described in What is F1 Optimal Threshold? How to calculate it?.

For a classifier that outputs a probability I would select the optimal F1 threshold on the validation set by examining the threshold that yields the best F1. This seems reasonable as selecting the threshold seems similar to selecting the best model, which would also be done using the validation set.

However, assume I have a classifier that does not output probabilities (like an SVM). How would you optimize the F1 on the validation set then?

$\endgroup$
  • $\begingroup$ Any updates on this? $\endgroup$ – Archie Nov 20 '18 at 10:30
0
$\begingroup$

In cases in which you have a classifier which simply outputs, say, a binary value, I think your best bet is to do hyper-parameter optimization on the validation set and then pick the set of parameters which maximize your F1 score. You could choose the parameters via cross-validation - you can be as sophisticated as you want there.

In general cases in which a classifier outputs at least a score, you can turn it into a probability by normalizing it. By the way, since you cited it, SVM use the Platt Scaling (article, wiki page, related question) to return a probability.

$\endgroup$
0
$\begingroup$

I think that you should use the approach that some framework uses (such as H2O), in the sense that you do not optimize F1, but you simply run a-posteriori research of the optimum threshold (for binary classification is the x where if prediction_proba > x then class 1, else class 0) used to determine the predicted class.

In this case, you can find the optimum to minimise misspredictions, simply calculating a F1 (or F_beta) score with input :

    - Y_true ( 0/1 for example)
    - ( Y_prediction_probabilities > threshold ) (0 or 1, will change when changing threshold )
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.