Standard ways to automatically remove incorrectly classified observations from a (mostly categorical) training dataset?

Question

I need a model that, given a (mostly categorical) labelled training data, cleans it up, removing incorrectly classified observations. What are the standard techniques to automatically detect and remove those observations?

For the purpose of model selection, I have synthetic labelled data whose incorrectly classified instances are known.

I found one algorithm to automatically detect outliers, the Isolation Forest, but looking at the documentation of its implementation in sklearn it seems to only care about X i.e. the distribution of input features, and not the labels y, which is not what I need. So I tried to incorporate y as a new feature of X to see if the detection worked, but this wasn't successful, and still ended up scoring only 17% accuracy in predicting whether an observation has an incorrect label.

Is there any other noise reduction algorithm I could use? Should I keep on with the Isolation Forest workaround and maybe give the feature y a heavier weight?

Answer 1

A simple approach that may be useful would be to run the anomaly detection separately for each class, to get X values that seem anomalous for a given class.

I would advise against using the binary decisions from the anomaly detection models (predict()). Both because tuning the decision threshold is tricky (and often ignored...). But also because for dataset curation you need to be careful to not automatically mess up your ground truth or introduce bias.

Instead I recommend using the anomaly scores (score_samples()) to sort the samples into a prioritized list. Then review these manually to check if the labels are correct or not. You should note down these decision, so you can track the efficiency of your sample selection. And you should compare this to a random selection. The reviewers should not know which selection method was in use. Or that you expect the labels to be wrong.