5
$\begingroup$

What does it mean to provide weights to each sample in a classification algorithm? How does a classification algorithm (eg. Logistic regression, SVM) use weights to give more emphasis to certain examples? I would love going into the details to unpack how these algorithms leverage weights.

If you look at the sklearn documentation for logistic regression, you can see that the fit function has an optional sample_weight parameter which is defined as an array of weights assigned to individual samples.

$\endgroup$
  • 1
    $\begingroup$ In most cases, it is as simple as weighing the loss function, such that more important (or perhaps rarer) observations contribute more strongly to the loss and visa versa. $\endgroup$ – Frans Rodenburg Aug 20 '18 at 8:36
  • $\begingroup$ the basic idea is just that the objective function is the sum of the loss on each sample, so it is easy to weight each sample differently, just as you calculate a weighted mean $\endgroup$ – seanv507 Sep 10 '18 at 17:05
3
$\begingroup$

As Frans Rodenburg already correctly stated in his comment, in most cases instance or sample weights factor into the loss function that is being optimized by the method in question.

Consider the equation the documentation provides for the primal problem of the C-SVM

$$\min_{w,b,\zeta} \frac{1}{2}w^Tw + C\sum_{i=1}^{n} \zeta_i. $$

Here $C$ is the same for each training sample, assigning equal 'cost' to each instance. In the case that there are sample weights passed to the fitting function

"The sample weighting rescales the C parameter, which means that the classifier puts more emphasis on getting these points right."

As this example puts it, which also provides a nice visualization, showing how instances represented by bigger circles (those with larger weight) influence the decision boundary.

enter image description here

$\endgroup$
  • $\begingroup$ Thanks for this awesome response! I like the visualization. $\endgroup$ – Jane Sully Sep 15 '18 at 14:32
  • $\begingroup$ I'd recommend you check out the scikit-learn user guide which contains a lot of great demos and explanations $\endgroup$ – deemel Sep 15 '18 at 21:18
3
$\begingroup$

Rickyfox's answer is great in explaining how the weights influence the results of a classifier, but maybe could you be also interested in why / how we would need such weights in the first place (which is more a statistical problem than a purely ML one).

Sometimes the observed data is observed with different distributions and we need to use sampling weights to account for it. You can look at Solon et. al (2015) for more details on why sampling weights matter for analyses and ML (uses mostly algorithms form econometrics literature, but the logic stays the same).

The idea is that these differences in distributions creates imbalances in classes and features. If untreated, this can affect the performance of the predictors / classifiers. I recently wrote a blog post about how you can use these weights to improve the accuracy of some algorithms (features an example with soccer data): https://nc233.com/2018/07/weighting-tricks-for-machine-learning-with-icarus-part-1/

The folowing image shows an example of feature imbalances: these teams of the dataset have not faced the same quality of opposition (elo). The prediction of the rarer types of matchups can be improved by reweighting techniques.

Example of feature imbalances: these teams of the dataset have not faced the same quality of opposition (elo). The prediction of the rarer types of matchups can be improved by reweighting techniques

Another example of good use of sampling weights is the treatment of class imbalances (typically when one of the classes is very rare). See for example what is done by default in scikit-learn: http://scikit-learn.org/stable/modules/generated/sklearn.utils.class_weight.compute_sample_weight.html

Finally, despite all these statistical reasons, sometimes we just need to "manually" increase the importance of an observation for very good reasons, and we use the weights to do so :)

References

Solon, Gary, Steven J. Haider, and Jeffrey M. Wooldridge. "What are we weighting for?." Journal of Human resources 50.2 (2015): 301-316.

$\endgroup$
  • 1
    $\begingroup$ Thank you for your response! Super helpful. I will definitely look into your blogpost. $\endgroup$ – Jane Sully Sep 15 '18 at 14:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.