6
$\begingroup$

I am working with some data that has been classified by domain experts. However, the classification they use is not 100% accurate. How can I deal with data that may not be correctly classified?

Are there any models which are more resilient to data that are not classified with 100% accuracy? Does such a classification bias entail limitations on the performance of a trained model?

$\endgroup$
2
  • $\begingroup$ have you used the cross validation . $\endgroup$
    – WOW
    Commented Apr 25, 2012 at 13:06
  • 1
    $\begingroup$ Cross Validation has nothing to do with this problem. $\endgroup$
    – Erik
    Commented Apr 25, 2012 at 14:02

2 Answers 2

2
$\begingroup$

Yes, there is a bias. For example, assume your classificator agrees with the expert 80% of the time. Now, there are several options, here are the two extremes: your model is better because the 20% where it does not agree is where the experts are wrong -> your performance is underestimated OR the 20% where you disagree are all cases where the experts are right -> your performance is overestimated.

You can find more info by searching for "imperfect gold standard". There are some nice bayesian methods availabe, but I am not familiar enough with them to recommend any. It might also more of a "multiple reader" problem, especially if your experts disagree with each other.

And, yes, your model will suffer if you train it with partly wrong class labels. It will try to emulate the flawed experts.

I don't know whether any particular method is particular resistant, I think a classificator that outputs a class probability could perform somewhat because you can correct somewhat for a expert bias toward one class by adjusting the cutoff. But that's just my intuition talking.

$\endgroup$
2
$\begingroup$

We have done some work on this for the case of random label flipping noise. Papers:

J. Bootkrajang and A. Kaban. Label-noise Robust Logistic Regression and its Applications. Proc. ECML-PKDD(1) 2012, pp. 143-158.

J. Bootkrajang and A. Kaban. Classification of Mislabelled Microarrays using Robust Sparse Logistic Regression. Bioinformatics. 29(7): 870-877, 2013.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.