Label importance scale - Supervised learning

Question

When using ML algorithms that optimize through some variation of gradient descent, I was wondering whether the scale of the label importance (weight) makes any difference.

Let's say we have a very unbalanced dataset (multiclass problem), and we want to give different weights to different training samples, dependent on their commonality in the dataset. Keeping the proportion of each weight relative to its commonality, how should I scale those weights? I'm pretty sure that weights 10-20-90 (class1-class2-class3) is very different from the perspective of the gradient than 0.1, 0.2, 0.9.

Should the weights have a mean of 1? Should they scale between 0..1? Any heuristic about this?

Answer 1

Depending on the optimization algorithm used, scaling can be an issue. With stochastic gradient descent, using a weight of 10/20/90 versus 0.1/0.2/0.9 is effectively the same as multiplying/dividing the learning rate by 100.

However, there are variants on SGD such as Adagrad and RMSProp which will adapt to the gradient size. This may mostly mitigate the effects of the loss scaling.

I usually follow the heuristic of keeping the final total loss value between 1 and 10, and it behaves well with the optimization algorithms. So if you scale the weights to be between 0 and 1, it will probably result in some reasonable total loss.