# Does threshold selection of F1-score in Cross-validation lead to overfitting?

I have a highly imbalanced binary classification problem. Right now I perform a 10-fold cross-validation while training my model (Convolutional Neural Network). Each fold generates its own F1-score, then I average all 10 F1-scores to produce the mean F1-score.

The question is if I select an optimal threshold for each fold's F1-scores, and then find the average of all F1-scores (which should definitely give me a better result in comparison to the threshold=0.5), would that be considered over-fitting? Because I already looked at the labels (precisions and recalls at different thresholds) while choosing a threshold, and then "chose" the most optimal F1-score.

Additionally, I didn't do any test set split. I assume 10 repetitions of 10-fold CV should be a good approximation to the test set as it is difficult to overfit in this case. The final model prediction at test-time would either be an average of 10 models, or a single model trained on the whole dataset. Not sure which option is better.

• I guess, the best approach is to select a threshold on 9 training folds. Sep 24, 2017 at 7:20
• Is there any reason why you have no test set? Sep 24, 2017 at 7:34
• Well, since it is pretty difficult to overfit on 10-fold CV, I thought it would be better to use all the data for training (and not utilize a test set), moreover a test set would be pretty small anyway, meaning it wouldn't be an accurate approximation of the population distribution. Sep 24, 2017 at 19:06
• Why use an improper scoring rule like $F_1$ instead of a proper scoring rule like log loss (which you already optimize, most likely, by using binary_crossentropy (that is the Keras syntax))?
– Dave
Jun 18, 2021 at 13:51