I'm using scikit-learn's RandomForest to perform a multi-class classification task, with examples from N classes and "garbage" examples not from the N classes.
Because the garbage examples might contain very different data, I'm not comfortable with just labelling them with one "garbage class". It intuitively (I'm not an expert) seems wrong to me to try to build a model for such a class because its instances will not be "clustered" in the feature space I'm using. But I still want to use them to get more robust models for my N classes.
So I'm thinking about running a one-vs-the-rest classification using OneVsRestClassifier to get N+1 classifiers (N + the garbage class), and only use the probabilities obtained for my N classes from the N classifiers to make the prediction. The prediction would be the class with the best probability above a threshold, or garbage if the best probability is lower than the threshold.
Does it makes sense to do N one-vs-the-rest classifiers and set a threshold on the probabilities obtained instead of just one classifier to predict the N+1 classes?
Is there any better way to deal with garbage data?