0
$\begingroup$

I need a model that, given a (mostly categorical) labelled training data, cleans it up, removing incorrectly classified observations. What are the standard techniques to automatically detect and remove those observations?

For the purpose of model selection, I have synthetic labelled data whose incorrectly classified instances are known.

I found one algorithm to automatically detect outliers, the Isolation Forest, but looking at the documentation of its implementation in sklearn it seems to only care about X i.e. the distribution of input features, and not the labels y, which is not what I need. So I tried to incorporate y as a new feature of X to see if the detection worked, but this wasn't successful, and still ended up scoring only 17% accuracy in predicting whether an observation has an incorrect label.

Is there any other noise reduction algorithm I could use? Should I keep on with the Isolation Forest workaround and maybe give the feature y a heavier weight?

$\endgroup$
2
  • $\begingroup$ What are the nature of your issues? Do you have labels y=A which should actually have been y=B? Or you want to drop out-of-distribution / data from classes that should not be included? $\endgroup$
    – Jon Nordby
    Mar 30, 2022 at 9:10
  • $\begingroup$ Since this is dataset curation I would advise caution and go through manually as practically possible, to ensure that the labels are correct. And if you need to automate things, it should probably be automation of the things you would do to manually correct the labels - not a generic automagical fixer model $\endgroup$
    – Jon Nordby
    Mar 30, 2022 at 9:12

1 Answer 1

0
$\begingroup$

A simple approach that may be useful would be to run the anomaly detection separately for each class, to get X values that seem anomalous for a given class.

I would advise against using the binary decisions from the anomaly detection models (predict()). Both because tuning the decision threshold is tricky (and often ignored...). But also because for dataset curation you need to be careful to not automatically mess up your ground truth or introduce bias.

Instead I recommend using the anomaly scores (score_samples()) to sort the samples into a prioritized list. Then review these manually to check if the labels are correct or not. You should note down these decision, so you can track the efficiency of your sample selection. And you should compare this to a random selection. The reviewers should not know which selection method was in use. Or that you expect the labels to be wrong.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.