logic behind balancing? [duplicate]

Question

I am a newbie in stats, and while reading: https://towardsdatascience.com/having-an-imbalanced-dataset-here-is-how-you-can-solve-it-1640568947eb

I don't seem to understand why is an imbalanced dataset wrong? And why should we rebalance…

I imagine this example: let’s say on a classification problem, for a very rare disease, where we have 1M rows for ‘no’ and only one row for ‘yes’.

My way of thinking is that, the dataset is unbalanced for a reason, (it’s a rare disease!) then wouldn’t balancing it remove that knowledge we have about it being rare?? I feel like by balancing we are discarding a very important piece of information…

(also I thought this was a vital piece of the bayesian world, 'the prior', aren't we removing the prior when we balance?)

Answer 1

I will answer from practical perspective.

Imbalanced dataset is not wrong, but it just requires strategy to handle and one of the strategy to handle that is rebalancing by oversampling or undersampling. You can think of balanced dataset to be a nice property to have, but even if it is not the case it would be okay.

The issue with this is actually lies on the machine learning side. Not all model "behaves" well under extreme imbalance classes. Supposed we used your example and assume we also do not apply any other strategy. What most likely to happen is that the model will predict "NO" for all cases and the loss from this would already be numerically small.

Now I have mentioned loss, another strategy to handle imbalanced dataset is using positive sample weight scaling, which will scale the loss on positive sample (minority), this works but when the scale constant is quite big it could lead to unstable training result. For example for an NN, supposed you use the last method, you can think of it like this, you would rarely see a positive sample, but as soon as you see the sample it would alter the weight by huge margin.

From all the above, You could see how having a balanced dataset would be a nice property to have. You can put equal weights on the samples without incurring any instability. For your example people would often apply other methods such as methods for anomaly detection. Again this answer is from practical perspective.