Why would be too optimistic to compute presicion, recall and f1-score to evaluate a model trained for imbalanced classification on a balanced testing sample ?

  • $\begingroup$ Do you have a source for this claim? $\endgroup$ – user2974951 Sep 5 '19 at 13:17
  • $\begingroup$ I am not sure if my claim is true in general. I just have few examples where metrics that are higher on a balanced testing sample than a imbalanced testing sample $\endgroup$ – Hector Blandin Sep 5 '19 at 13:27
  • $\begingroup$ I don't think it's optimistic at all to build a model on an imbalanced data set and then evaluate it on a balanced data set. Or the other way around. It depends on the data. $\endgroup$ – user2974951 Sep 6 '19 at 6:58

I do not quite understand what you mean by “optimistic”. Training on a balanced set and testing on an imbalanced set is fine to me, as long as the test set model the real distribution of the data well and the classifier performs well.

However, if you want to estimate the precision on the imbalanced set based on the performance on the training set, that will not work. Precision and recall will look very different between balanced and imbalanced set.

| cite | improve this answer | |
  • $\begingroup$ opimistic in the sense of having high metrics like f1-score, presicion and recall. But in reality, the model maybe will not generalize well. $\endgroup$ – Hector Blandin Oct 17 '19 at 17:00
  • 1
    $\begingroup$ In my opinion, when your training set and test set have very different class distribution, you cannot compare performance metrics that are sensitive to class distribution to detect over fitting. However, performance on test set still can be a estimate of how well the model will perform on unseen data. $\endgroup$ – etudiant Oct 18 '19 at 5:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.