3
$\begingroup$

Let's consider using mini-batch SGD in (neural network) binary classification problem with imbalanced dataset. Let's say that the ratio between the number of examples in each class is positive:negative=1:99. Let's say I want to balance the dataset. Then, there are two options: upsample the minority class OR add weights to the loss function. I often hear that both are equivalent. However, I think using class weights make the minimum search more stochastic and probably leads to less stable results.

Consider a mini-batch SGD update:

sgd_update

where L is the loss function, eta is the learning rate, theta are current model parameters, X are our mini-batch samples.

Now we take the weights that are inverse proportional to the number of examples, s.t. the sum of weights equals the number of classes. This gives w_0=0.02 for the negative class and w_1=1.98 for the positive class.

Now the mini-batch SGD update becomes:

sgd_update2

where theta_Lpos is the loss computed for positive-class examples and theta_Lneg is the loss computed for negative-class examples in the minibatch.

So, the effective learning rate for classes changes proportional to class weights. In particular, the learning rate for the majority class sees a 50x drop!

Now let's say we train in mini-batches of N_batch = 16. If we create our minibatches by randomly taking samples from the dataset, most of the minibatches will consist of negative examples only. As a result, the change in model parameters due to such a minibatch will be extremely low as the learning rate is leveraged by the too small class weight. With such a low learning rate there is a risk to get stuck in a local minimum of theta_Lneg. despite training in batches.

However, in minibatches where there is at least one positive example the loss function will be dominated by the contribution of the positive examples due to large w_1. This can kick us out of the local minimum trap and move closer to the minimum of loss function for positive examples theta_Lpos.

What happens when we do upsampling instead of placing class weights?

The number of positive examples gets equal to the number of negative examples, no need to impose class weights, the learning rate is adequate for both classes and in each minibatch there are as many positive examples as negative. At each SGD update both positive and negative examples are considered and we gradually move towards the true minimum of theta_L instead of sporadically exploring the parameter search space by jumping between the minima of theta_Lpos and theta_Lneg.

So, it seems upsampling should lead to faster convergence and less variation in final results than using class weights. Please share your thoughts.

$\endgroup$

1 Answer 1

-1
$\begingroup$

Did you hear about class scaling and WeightedRandomSampler. I think if your data set is 1:10 class imbalance, it is fine to use class weight or upsampling. 1:99 is not class imbalance problem, it is kind of outlier detection. In term of model convergernce, I agree with your idea, but it really depends the data, the model.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.