How to deal with garbage data with Random Forest?

Question

I'm using scikit-learn's RandomForest to perform a multi-class classification task, with examples from N classes and "garbage" examples not from the N classes.

Because the garbage examples might contain very different data, I'm not comfortable with just labelling them with one "garbage class". It intuitively (I'm not an expert) seems wrong to me to try to build a model for such a class because its instances will not be "clustered" in the feature space I'm using. But I still want to use them to get more robust models for my N classes.

So I'm thinking about running a one-vs-the-rest classification using OneVsRestClassifier to get N+1 classifiers (N + the garbage class), and only use the probabilities obtained for my N classes from the N classifiers to make the prediction. The prediction would be the class with the best probability above a threshold, or garbage if the best probability is lower than the threshold.

Does it makes sense to do N one-vs-the-rest classifiers and set a threshold on the probabilities obtained instead of just one classifier to predict the N+1 classes?

Is there any better way to deal with garbage data?

Answer 1

I do not see any advantage in using OneVsRestClassifier over using RandomForest directly. In OneVsRest at each step (that is, for each class as "positive" class) you are labeling as "rest" not only your 'garbage', but the other classes as well. You write

Because the garbage examples might contain very different data, I'm not comfortable with just labelling them with one "garbage class".

I think, using OneVsRest will make this problem only worse. In addition, at each step you are artificially making your dataset imbalanced, which introduces extra trouble.

Generally, I always considered OneVsRest as a "poor man's substitute" (or whatever the politically correct expression is) used for a classifiers which are intrinsically two-class, like SVC. Please, correct me, if I am wrong. RandomForest is intrincically multiclass and this is its advantage.

Further, since RandomForest is a supervised method, your 'garbage' examples are labeled somehow from the very beginning, correct? If there is one 'garbage' label from the very beginning, you can hardly do anything about it. If there are several labels, why not to keep them all? If there are too many labels, you can try to combine/group them. How? Several ideas are possible

• use your inside business knowledge

• try to keep dataset as balanced as possible/reasonable

• use some clustering method for 'garbage' and make new labels accourding to clusters

Answer 2

It is hard to tell IMO. Depending on the data, you might observe different results.

One thing I can suggest is instead of one-vs-one, you may want to try a classifier for each pair.

Regarding garbage data, one way to go is to apply clustering over the garbage data with number of clusters set by rule of thumb. This may improve your overall classification performance given that the garbage data you see in test follows similar distribution.