I am using Boosted Decision Tree Regression to predict a value. I get a few values where the residual (True Value - Estimated Value) in my training and test sets are large. I have a reason to believe that these are recording errors. The few I can spot by eye makes me concerned that there are others which I can't pick out so easily. A boosted decision tree is very sensitive to outliers so I would like to remove them from my training set before I train it.
My first thought was to use a Random Forrest because it is a similar method which is less sensitive to outliers and will also estimate the standard deviation in addition to the dependent variable. I could then remove all training samples with a residual greater than a few standard deviations. However, this method did not work. It seems that the standard deviation is calculated such that it will be higher when it is estimating for an outlier. This makes sense since I am not using a training and testing set. If I do this, the method works better but then I greatly reduce my data for the eventual prediction algorithm. I was considering using Quantile Regression Forests and then cut off the 5 and 95 percentiles but preliminary tests look like it is too slow.
Are there any standard methods for this?
I have made progress on my original idea. ie Run a random forest on the training set to see which values are difficult to estimate, then remove them before training with the Boosted Decision Tree. To illustrate the issue better I have added a number of plots below. They are all scatter plots of the various metrics:
- The true value: The value which I want to estimate
- The Estimated Value: The output of the random Forrest made by averaging the predictions from all the individual regression trees.
- The Residual: true value - Estimated Value
- The STD: An estimate of the uncertainty of the prediction made as the standard deviation of the predictions from all the individual regression trees.
The recording errors that can be seen by eye have values around 90,000 and 100,000. As said before the presence of these recording errors makes me suspect that there are others which are not so easy to spot.
I am considering cuts on two possible things to reduce the measurement errors. The first is the STD alone, where I would cut anything over 20000. The second is the Studentized residual (Res/STD). I show this in all the plots grouped by value in colours. A cut at about 3 seems reasonable.
The issue with this method is that the values of the metrics I want to cut on are highly dependent on the tuning of the hyperparameters. Likely the most crucial is the "Minimum number of samples per leaf node". I have set this to 10 which is sort of high. The idea being that the records with recording errors will be easily grouped with other values based on the features but have a very different value. Forcing this group to contain at least 10 values should make the outlier clear.