Sample weighting vs. (e.g. one-hot encoded) categories

Question

I have seen recommendations to use sample weighting when the training dataset is not evenly balanced over known categories, so that an imbalance in the number of elements in each category does not skew the training towards the most frequently represented types (i.e. the #samples x sample weights is made the same for all categories)

If, however, training specifically includes the categories (and within which there is either no further skewing effect or it is immaterial) is it necessary or desirable to include weighting as well?

Naively, one might assume that one-hot encoded categories effectively switch between different parameter sets.

However, the extent to which the input 1's and 0's are then reflected in internal 1, 0 weights is surely a function of the degree of training.

In other words, is it true that, whilst in theory sample weighting over categories is not required as #epochs -> infinity, in practice, sample weighting ensures that the model is effectively as good as it can be for #epochs < infinity, even when the weighting is over categories explicitly included as input?

Update

Thanks to @Tim's linked paper, in section 1.1 we find:

Despite the popularity of importance sampling in combination with deep neural networks, how and when it works remain open questions. Unlike linear models, deep neural networks are generally over-parameterized, capable of fitting training datasets to perfect accuracy (Zhang et al., 2017). Moreover, it is now recognized that for many tasks deep neural networks continue to improve generalization error past the point of achieving zero training error (Soudry et al., 2017). Thus they are not only capable of separating the training set (given enough epochs) but actually are trained to do so in common practice. Since neural networks are capable of shattering the training set (and often do), it is not clear that any trade-offs must be made among classifying each of the training points. Thus any effects of importance weighting depend crucially on how they impact the dynamics of optimization, an actively-studied but still poorly-understood topic.

(emphasis added)

It seems that the authors are of similar opinion to me; I suspect that because nn's can shatter* the dataset, whether or not they do so perfectly at any time depends on lots of things, but in general the shattering can be perfected given sufficient training (provided in practice one doesn't run out of precision)

Definition: A set S of examples is shattered by a set of functions H if for every partition of the examples in S into positive and negative examples there is a function in H that gives exactly these labels to the examples Intuition: A rich set of functions shatters large sets of points (from ...Shattering and VC Dimensions)

But - is there anything else to add to the idea?

It seems that the impact of importance weighting diminishes, but if one has limited time one would get better results by including it.

Answer 1

There was an interesting paper What is the Effect of Importance Weighting in Deep Learning? by Byrd and Lipton, where they've shown the following:

1. We demonstrate the surprising finding that for regularized neural networks optimized by stochastic gradient descent (SGD), the impact of importance weighting diminishes over epochs of training.
2. We show that L2 regularization and batch normalization, (but not dropout) interact with importance weights, restoring (some) impact on learned models.
3. We replicate our results across a variety of networks, tasks, and datasets.
4. Our results call into question the standard application of importance weighting when applied to deep networks, a finding with practical consequences on the fields of causal inference, domain adaptation, and off-policy reinforcement learning.

So it works for traditional machine learning models (say, logistic regression), there are mixed results when using it for neural networks. Notice that the results show something opposite to what you suggested. Moreover, this has nothing to do with one-hot-encoding.