When using ML algorithms that optimize through some variation of gradient descent, I was wondering whether the scale of the label importance (weight) makes any difference.
Let's say we have a very unbalanced dataset (multiclass problem), and we want to give different weights to different training samples, dependent on their commonality in the dataset. Keeping the proportion of each weight relative to its commonality, how should I scale those weights? I'm pretty sure that weights 10-20-90 (class1-class2-class3) is very different from the perspective of the gradient than 0.1, 0.2, 0.9.
Should the weights have a mean of 1? Should they scale between 0..1? Any heuristic about this?