Assume I have a dataset split into train/val/test and I want to compute the optimal threshold value for an F1 score. This threshold value is in [0, 0.5] as described in What is F1 Optimal Threshold? How to calculate it?.
For a classifier that outputs a probability I would select the optimal F1 threshold on the validation set by examining the threshold that yields the best F1. This seems reasonable as selecting the threshold seems similar to selecting the best model, which would also be done using the validation set.
However, assume I have a classifier that does not output probabilities (like an SVM). How would you optimize the F1 on the validation set then?