I have an imbalanced dataset; 95% negative class and 5% positive class. I split my data into train (80%) and test (20%) sets. I am using 5-fold cross-validation on the train set to determine the optimal hyperparameters. I'm unsure how to determine the optimal threshold with cross-validation.
Does the following process make sense:
- loop through each fold
- train the model
- determine the optimal threshold on the train set
- calculate the f1 score on the held-out set using the threshold obtained from step 3.
The above process leads to 5 thresholds. I select the threshold with the best f1 score on the hold-out sets. Lastly, finalize the model assessment on the test set.