A friend of mine has code, which rebalances the classes of the test set before running the algorithm and calculating the accuracy. This causes the distribution of the two classes to be 50%/50% instead of 35%/65%. Is that problematic?

One class may be harder to predict than the other and when rebalancing the classes, it can change the test accuracy from the real accuracy. However, I'm not quite sure whether it's problematic and whether it would inflate/deflate accuracy?

EDIT: Tried to clarify my question based on comments.

  • 2
    $\begingroup$ What exactly do you mean by "rebalancing"? $\endgroup$ – cbeleites supports Monica Apr 16 '15 at 7:51
  • 1
    $\begingroup$ I am assuming that your friend doesn't set a seed during the creation of the training/validation sets, which results in different results every time they run the code? Tell them to set the seed so that his results are reproducible! $\endgroup$ – Matt Reichenbach Apr 16 '15 at 12:59
  • $\begingroup$ I've updated the question. I don't know about the seed, but that's an important point! $\endgroup$ – pir Apr 16 '15 at 13:55
  • 2
    $\begingroup$ Touching the test set is absolutely forbidden, and the accuracy measured on a 'rebalanced' test set should not be trusted. The rule is that you can do pretty much everything you want to your model or data at training time (including over or undersampling from your classes), but final evaluation should be conducted on the original test set. Actually, if you modify your test set, you modify the task, so you are not training and evaluating your model for the intended task anymore. $\endgroup$ – Antoine May 16 '17 at 17:56

I don't agree on doing that for test group since which is the last step before moving to predict the real world that without "true" cases. The accuracy we got in that way therefore cannot imply the accuracy we could say for the upcoming unknown events. Even though I heard people talking this a lot but I just personally disagree on using it for the test group. Please justify my answer if you know what's the true right answer here.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.