1
$\begingroup$

Suppose I want to get a model, such as a neural network, to correctly classify pictures of cats and dogs and I know that the test set contains around $1\%$ of cats and $99\%$ of dogs.

My intuition is that if I have very few training data, it would be better to use a training dataset that is highly unbalanced. This is because with few data the model doesn't stand a chance trying to 'understand' what is a cat or a dog, and in this case it would work better by being highly biased towards the highly represented class. It would simply learn to assume a target distribution and behave accordingly.

My intuition also says that, on the opposite, if I had a very large training set there would be no need to use an unbalanced train set, since the model can 'understand' the properties of each class and does not need to rely on a priori assumptions on the distribution of the test dataset.

So the question could be, if I used, let's say, $50$-million pictures of cats and dogs each for training set, would I get a worse result than if I presented it with $99$-million dogs and $1$-milliom cats? Also, I believe that this would very likely depend on the model used (I assume the answer for a neural network may differs from the answer for SVM).

Improve this question
$\endgroup$
4
  • $\begingroup$ I do wonder if having a gigantic number of observations allows the signal to scream out over the noise, almost regardless of prior probability. $\endgroup$
    – Dave
    Commented Dec 5, 2022 at 0:28
  • $\begingroup$ This all depends on how the model is constructed. If it is a generative classifier (e.g. a parametric Gaussian classifier) then the models of the distribution of patterns in each class are constructed independently, so it won't be affected by the imbalance. Similarly, if a discriminative classifier only learns about one class, it will not have minimised the cost function, so if it doesn't learn the minority class, it is in a local minima - which can't happen if the cost function is convex. So this is all classifier dependent. $\endgroup$
    – Dikran Marsupial
    Commented Apr 15, 2023 at 7:26
  • $\begingroup$ Note the optimal decision boundary depends on the class ratio, so if you balance the training set, the classifier is likely to over-predict the minority class, so you would have to correct for that. $\endgroup$
    – Dikran Marsupial
    Commented Apr 15, 2023 at 7:27
  • $\begingroup$ I posted a question asking for examples where rebalancing improves accuracy (stats.stackexchange.com/questions/559294/…) and there were no answers, even when there was a modest bonus, which suggests that this is not a real problem in practice. $\endgroup$
    – Dikran Marsupial
    Commented Apr 15, 2023 at 7:28

1 Answer 1

Reset to default
1
$\begingroup$

I would expect the issue to go away as you got an enormous amount of data, especially if you used a flexible model that can figure out patterns (e.g., deep learning). You can think of the probability prediction in terms of Bayes' theorem, where you have a prior probability (the class ratio) and get the posterior probability as your prediction. As the data size gets large, the prior probability ought to get overwhelmed by the data, allowing an unusual case to scream out as being particularly likely to belong to the minority category, regardless of prior probability. Thus, whether you do artificial balancing or not, the model should figure out the truth. While you do not have to go full-Bayesian to view the problem in these terms, it might help you to think of how even a rather strong prior distribution gets overwhelmed by a huge amount of contradictory data in maximum a posteriori estimation.

(Now, it might be that such an event is unusual no matter what, but a good model trained on a huge data set should catch this.)

Improve this answer
$\endgroup$
1
  • $\begingroup$ +1 where imbalance is a genuine problem, it is because of estimation problems. Adding more data solves most estimation problems eventually! $\endgroup$
    – Dikran Marsupial
    Commented Apr 15, 2023 at 7:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.