Random forest retraining on only correctly predicted records

Question

A colleague of mine suggested the following procedure for a categorization problem: 1) Train a random forest 2) Select only the records from the training data set which the random forest predicted correctly (based on in sample predictions) 3) Train a random forest on this subset 4) Use this random forest to make predictions for new data (out of sample)

He claims that this procedure is helpful for removing noise from the data as wrongly predicted records in the training sample are likely to be noise. However, I think that this is not a good procedure because I cannot imagine how predictive capability of the model should improve by excluding wrongly predicted data. We just retrain a new random forest on data we could already predict correctly with the first one, and we cannot know whether the wrongly predicted records are really noise or whether our model just didn't capture them correctly. Furthermore, I haven't heard or read anything about such an approach before.

Does anybody have any thoughts on this? I'd also highly appreciate any hints to papers in which this procedure has been applied before.

Answer 1

A good example which illustrates why this is problematic is class imbalance. Suppose we have 10 features of class 1 and 90 features of class 2. A default model is likely to be biased towards the more representative class, misclassifying more cases of class 1 to optimize overall classification success (so careful consideration of class imbalance is essential to improve classification f or both classes). Removing all of those misclassified cases is the opposite of what you would want to do, as this would make the model even less likely to predict class 1 correctly. To generalize, this would further increase error rates for the class which had the most misclassified samples.

In the unlikely case of misclassified samples equi-distributed across classes (for which you may want to try a study-specific simulation of the effect of this procedure if you have theoretical interest), this is still not advisable as it would be an equivalent of dropping 'bad' points from a linear regression - i.e. 'alternative data' if you wish.