I have balanced training data: 300 positives and 300 negatives, but the test data is unbalanced: I have 15 positives and 60 negatives. Will the unbalanced test data impact classification accuracy?

Actually, I tried to collect balanced test data (30 pos & 30 neg). The precision on positives are better. Is the reason the balance of test data? Thanks.


Depends entirely on what your model's properties and the performance measure are.

Lets take a simple example: suppose your model predicts all positives and half of the negatives correctly on your training set (e.g. accuracy 75%). Assuming the positives and negatives in the test set have similar distributions to those in the training set, the model will probably perform similarly on the test set. In that case the accuracy would drop to $(15+30)/75 = 60\%$.

You have to answer two questions before looking at performance numbers:

  1. What properties of the model matter to you?
  2. What score measure do you use to capture (1)?

You can eliminate the effect of unbalanced data by using score measures such as area under the ROC curve.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.