# Better understanding classification with unbalanced test data from a mathematical perspective

Suppose I want to get a model, such as a neural network, to correctly classify pictures of cats and dogs and I know that the test set contains around $$1\%$$ of cats and $$99\%$$ of dogs.

My intuition is that if I have very few training data, it would be better to use a training dataset that is highly unbalanced. This is because with few data the model doesn't stand a chance trying to 'understand' what is a cat or a dog, and in this case it would work better by being highly biased towards the highly represented class. It would simply learn to assume a target distribution and behave accordingly.

My intuition also says that, on the opposite, if I had a very large training set there would be no need to use an unbalanced train set, since the model can 'understand' the properties of each class and does not need to rely on a priori assumptions on the distribution of the test dataset.

So the question could be, if I used, let's say, $$50$$-million pictures of cats and dogs each for training set, would I get a worse result than if I presented it with $$99$$-million dogs and $$1$$-milliom cats? Also, I believe that this would very likely depend on the model used (I assume the answer for a neural network may differs from the answer for SVM).

• I do wonder if having a gigantic number of observations allows the signal to scream out over the noise, almost regardless of prior probability.
– Dave
Dec 5, 2022 at 0:28
• This all depends on how the model is constructed. If it is a generative classifier (e.g. a parametric Gaussian classifier) then the models of the distribution of patterns in each class are constructed independently, so it won't be affected by the imbalance. Similarly, if a discriminative classifier only learns about one class, it will not have minimised the cost function, so if it doesn't learn the minority class, it is in a local minima - which can't happen if the cost function is convex. So this is all classifier dependent. Apr 15, 2023 at 7:26
• Note the optimal decision boundary depends on the class ratio, so if you balance the training set, the classifier is likely to over-predict the minority class, so you would have to correct for that. Apr 15, 2023 at 7:27
• I posted a question asking for examples where rebalancing improves accuracy (stats.stackexchange.com/questions/559294/…) and there were no answers, even when there was a modest bonus, which suggests that this is not a real problem in practice. Apr 15, 2023 at 7:28