3
$\begingroup$

I have a highly imbalanced binary classification problem. Right now I perform a 10-fold cross-validation while training my model (Convolutional Neural Network). Each fold generates its own F1-score, then I average all 10 F1-scores to produce the mean F1-score.

The question is if I select an optimal threshold for each fold's F1-scores, and then find the average of all F1-scores (which should definitely give me a better result in comparison to the threshold=0.5), would that be considered over-fitting? Because I already looked at the labels (precisions and recalls at different thresholds) while choosing a threshold, and then "chose" the most optimal F1-score.

Additionally, I didn't do any test set split. I assume 10 repetitions of 10-fold CV should be a good approximation to the test set as it is difficult to overfit in this case. The final model prediction at test-time would either be an average of 10 models, or a single model trained on the whole dataset. Not sure which option is better.

$\endgroup$
  • $\begingroup$ I guess, the best approach is to select a threshold on 9 training folds. $\endgroup$ – Anuar Y Sep 24 '17 at 7:20
  • $\begingroup$ Is there any reason why you have no test set? $\endgroup$ – lnathan Sep 24 '17 at 7:34
  • $\begingroup$ Well, since it is pretty difficult to overfit on 10-fold CV, I thought it would be better to use all the data for training (and not utilize a test set), moreover a test set would be pretty small anyway, meaning it wouldn't be an accurate approximation of the population distribution. $\endgroup$ – Anuar Y Sep 24 '17 at 19:06
2
$\begingroup$

A proper procedure always includes training - validation and test set. A small dataset is usually not a good reason to not include a test set, as only this will give you a reasonable estimation of your model.

Your approach does not necessarily lead to overfitting. But you will not notice if that happens. How usefull is a model, that does not generalize enough? Especially in a highly imbalanced classification problem, you should hold out a test set.

Notice that in k-fold CV every instance is predicted exactly once, so in fact you do not "average" anything here. You just kind of "merge" your predictions of each fold.

After you select your threshold, tune parameters etc. you want to use all of your data to train a final model to get the best possible results.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.