Let's consider using mini-batch SGD in (neural network) binary classification problem with imbalanced dataset. Let's say that the ratio between the number of examples in each class is positive:negative=1:99. Let's say I want to balance the dataset. Then, there are two options: upsample the minority class OR add weights to the loss function. I often hear that both are equivalent. However, I think using class weights make the minimum search more stochastic and probably leads to less stable results.
Consider a mini-batch SGD update:
where L is the loss function, is the learning rate,
are current model parameters, X are our mini-batch samples.
Now we take the weights that are inverse proportional to the number of examples, s.t. the sum of weights equals the number of classes. This gives w_0=0.02 for the negative class and w_1=1.98 for the positive class.
Now the mini-batch SGD update becomes:
where is the loss computed for positive-class examples and
is the loss computed for negative-class examples in the minibatch.
So, the effective learning rate for classes changes proportional to class weights. In particular, the learning rate for the majority class sees a 50x drop!
Now let's say we train in mini-batches of N_batch = 16. If we create our minibatches by randomly taking samples from the dataset, most of the minibatches will consist of negative examples only. As a result, the change in model parameters due to such a minibatch will be extremely low as the learning rate is leveraged by the too small class weight. With such a low learning rate there is a risk to get stuck in a local minimum of . despite training in batches.
However, in minibatches where there is at least one positive example the loss function will be dominated by the contribution of the positive examples due to large w_1. This can kick us out of the local minimum trap and move closer to the minimum of loss function for positive examples .
What happens when we do upsampling instead of placing class weights?
The number of positive examples gets equal to the number of negative examples, no need to impose class weights, the learning rate is adequate for both classes and in each minibatch there are as many positive examples as negative. At each SGD update both positive and negative examples are considered and we gradually move towards the true minimum of instead of sporadically exploring the parameter search space by jumping between the minima of
and
.
So, it seems upsampling should lead to faster convergence and less variation in final results than using class weights. Please share your thoughts.
