Threshold optimization with cross validation

Question

I have an imbalanced dataset; 95% negative class and 5% positive class. I split my data into train (80%) and test (20%) sets. I am using 5-fold cross-validation on the train set to determine the optimal hyperparameters. I'm unsure how to determine the optimal threshold with cross-validation.

Does the following process make sense:

1. loop through each fold
2. train the model
3. determine the optimal threshold on the train set
4. calculate the f1 score on the held-out set using the threshold obtained from step 3.

The above process leads to 5 thresholds. I select the threshold with the best f1 score on the hold-out sets. Lastly, finalize the model assessment on the test set.

Answer 1

The correct and efficient steps should be as follows:

• loop through each fold
• train the model
• try each threshold value you want to test on the validation fold
• report the threshold that gives the best performance metric (in your case F1) when averaged over all folds

The only difference compared to a typical hyper-parameter is that you do not need to train the model for trying different thresholds.

I'd also like to note that the thresholds are not fully regarded as hyper-parameters of the underlying model as they do not change the model flexibility but only reflect a measure of your classifier's trade-off.