1
$\begingroup$

I have an imbalanced dataset; 95% negative class and 5% positive class. I split my data into train (80%) and test (20%) sets. I am using 5-fold cross-validation on the train set to determine the optimal hyperparameters. I'm unsure how to determine the optimal threshold with cross-validation.

Does the following process make sense:

  1. loop through each fold
  2. train the model
  3. determine the optimal threshold on the train set
  4. calculate the f1 score on the held-out set using the threshold obtained from step 3.

The above process leads to 5 thresholds. I select the threshold with the best f1 score on the hold-out sets. Lastly, finalize the model assessment on the test set.

$\endgroup$

1 Answer 1

0
$\begingroup$

The correct and efficient steps should be as follows:

  • loop through each fold
  • train the model
  • try each threshold value you want to test on the validation fold
  • report the threshold that gives the best performance metric (in your case F1) when averaged over all folds

The only difference compared to a typical hyper-parameter is that you do not need to train the model for trying different thresholds.

I'd also like to note that the thresholds are not fully regarded as hyper-parameters of the underlying model as they do not change the model flexibility but only reflect a measure of your classifier's trade-off.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.