How do sample weights work in classification models?

Question

What does it mean to provide weights to each sample in a classification algorithm? How does a classification algorithm (eg. Logistic regression, SVM) use weights to give more emphasis to certain examples? I would love going into the details to unpack how these algorithms leverage weights.

If you look at the sklearn documentation for logistic regression, you can see that the fit function has an optional sample_weight parameter which is defined as an array of weights assigned to individual samples.

Answer 1

As Frans Rodenburg already correctly stated in his comment, in most cases instance or sample weights factor into the loss function that is being optimized by the method in question.

Consider the equation the documentation provides for the primal problem of the C-SVM

$$\min_{w,b,\zeta} \frac{1}{2}w^Tw + C\sum_{i=1}^{n} \zeta_i. $$

Here $C$ is the same for each training sample, assigning equal 'cost' to each instance. In the case that there are sample weights passed to the fitting function

"The sample weighting rescales the C parameter, which means that the classifier puts more emphasis on getting these points right."

As this example puts it, which also provides a nice visualization, showing how instances represented by bigger circles (those with larger weight) influence the decision boundary.

Answer 2

Rickyfox's answer is great in explaining how the weights influence the results of a classifier, but maybe could you be also interested in why / how we would need such weights in the first place (which is more a statistical problem than a purely ML one).

Sometimes the observed data is observed with different distributions and we need to use sampling weights to account for it. You can look at Solon et. al (2015) for more details on why sampling weights matter for analyses and ML (uses mostly algorithms form econometrics literature, but the logic stays the same).

The idea is that these differences in distributions creates imbalances in classes and features. If untreated, this can affect the performance of the predictors / classifiers. I recently wrote a blog post about how you can use these weights to improve the accuracy of some algorithms (features an example with soccer data): https://nc233.com/2018/07/weighting-tricks-for-machine-learning-with-icarus-part-1/

The folowing image shows an example of feature imbalances: these teams of the dataset have not faced the same quality of opposition (elo). The prediction of the rarer types of matchups can be improved by reweighting techniques.

Another example of good use of sampling weights is the treatment of class imbalances (typically when one of the classes is very rare). See for example what is done by default in scikit-learn: http://scikit-learn.org/stable/modules/generated/sklearn.utils.class_weight.compute_sample_weight.html

Finally, despite all these statistical reasons, sometimes we just need to "manually" increase the importance of an observation for very good reasons, and we use the weights to do so :)

References

Solon, Gary, Steven J. Haider, and Jeffrey M. Wooldridge. "What are we weighting for?." Journal of Human resources 50.2 (2015): 301-316.