I have a highly imbalanced binary classification problem. Right now I perform a 10-fold cross-validation while training my model (Convolutional Neural Network). Each fold generates its own F1-score, then I average all 10 F1-scores to produce the mean F1-score.
The question is if I select an optimal threshold for each fold's F1-scores, and then find the average of all F1-scores (which should definitely give me a better result in comparison to the threshold=0.5), would that be considered over-fitting? Because I already looked at the labels (precisions and recalls at different thresholds) while choosing a threshold, and then "chose" the most optimal F1-score.
Additionally, I didn't do any test set split. I assume 10 repetitions of 10-fold CV should be a good approximation to the test set as it is difficult to overfit in this case. The final model prediction at test-time would either be an average of 10 models, or a single model trained on the whole dataset. Not sure which option is better.