Let us say I have labelled training data. P predictors, C classes, N exemplars. I also have one additional dimension (not calling it a predictor for now) which is the confidence of the label in the Class column.

Now what if I have some additional information regarding the confidence of the class labels. Perhaps I had more agreement by curators on the class label of a particular exemplar than another.

Is there a way to take advantage of this information?


Sure, but it depends on what kind of classifier you're using.

For most classifiers, it's relatively easy to add an instance weight to each example. For example, you might minimize $$\sum_{i=1}^n w_i \ell(f(x_i), y_i),$$ where $\ell$ is the loss function for each data point, and $w_i$ is some relative weight you've come up with to put more emphasis on the points whose labels you're more confident of.

You might also treat it probabilistically: assuming that a label is $1$ with probability $p_i$ and $0$ with probability $1-p_i$, and that these labels are independent of one another, you could minimize the expected loss as $$\sum_{i=1}^n p_i \ell(f(x_i), 1) + (1-p_i) \ell(f(x_i), 0).$$


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.