Outlier removal before Boosted Decision Tree Regression

Question

I am using Boosted Decision Tree Regression to predict a value. I get a few values where the residual (True Value - Estimated Value) in my training and test sets are large. I have a reason to believe that these are recording errors. The few I can spot by eye makes me concerned that there are others which I can't pick out so easily. A boosted decision tree is very sensitive to outliers so I would like to remove them from my training set before I train it.

My first thought was to use a Random Forrest because it is a similar method which is less sensitive to outliers and will also estimate the standard deviation in addition to the dependent variable. I could then remove all training samples with a residual greater than a few standard deviations. However, this method did not work. It seems that the standard deviation is calculated such that it will be higher when it is estimating for an outlier. This makes sense since I am not using a training and testing set. If I do this, the method works better but then I greatly reduce my data for the eventual prediction algorithm. I was considering using Quantile Regression Forests and then cut off the 5 and 95 percentiles but preliminary tests look like it is too slow.

Are there any standard methods for this?

EDIT

I have made progress on my original idea. ie Run a random forest on the training set to see which values are difficult to estimate, then remove them before training with the Boosted Decision Tree. To illustrate the issue better I have added a number of plots below. They are all scatter plots of the various metrics:

• The true value: The value which I want to estimate
• The Estimated Value: The output of the random Forrest made by averaging the predictions from all the individual regression trees.
• The Residual: true value - Estimated Value
• The STD: An estimate of the uncertainty of the prediction made as the standard deviation of the predictions from all the individual regression trees.

The recording errors that can be seen by eye have values around 90,000 and 100,000. As said before the presence of these recording errors makes me suspect that there are others which are not so easy to spot.

I am considering cuts on two possible things to reduce the measurement errors. The first is the STD alone, where I would cut anything over 20000. The second is the Studentized residual (Res/STD). I show this in all the plots grouped by value in colours. A cut at about 3 seems reasonable.

The issue with this method is that the values of the metrics I want to cut on are highly dependent on the tuning of the hyperparameters. Likely the most crucial is the "Minimum number of samples per leaf node". I have set this to 10 which is sort of high. The idea being that the records with recording errors will be easily grouped with other values based on the features but have a very different value. Forcing this group to contain at least 10 values should make the outlier clear.

Answer 1

Loss function

Random Forests (RF) andBBoosted Decision Trees (BDT) are techniques that optimize on a given loss function. In sklearn, the selection of loss function is done by configuring the ´criterion´ parameter. This should also very likely possible with the statistical tools available on Azure.

There are loss functions that are more or less sensitive to outliers, for example the MSE is more sensitive to outliers than the Mean Absolute Error (equivalent to estimating the conditional 50 percentile, aka the conditional median). You can see a rich portfolio of loss functions here. I see the choice of loss function a convenient way to assign more or less weight to "outliers".

Edit 1:

I have been working on speeding up the quartile regression

The estimation of quantiles can also be done by modifying the loss function so that, instead of estimating $E[Y>X]$ or the median (MAE) (2 quartile), it will generate a conditional estimation for an arbitrary quantile. This is done by minimizing a sum of asymmetrically weighted absolute residuals (see). While you say that you would not switch to a loss function which is less sensitive to measurement errors, you might have already done so.

Outliers vs high leverage points vs influential points

It is important to distinguish the concept of outlier from the concept of high leverage and influential points

• An outlier is a data point whose response y does not follow the general trend of the rest of the data.
• A data point has high leverage if it has "extreme" predictor x values.
• A data point is influential if it unduly influences any part of a regression analysis, such as the predicted responses, the estimated slope coefficients, or the hypothesis test results. Outliers and high leverage data points have the potential to be influential (source)

As you mention:

I could then remove all training samples with a residual greater than a few standard deviations. This method did not work.

I thought it would be useful to point you to strategies and techniques to deal with outliers, leverage and ultimately influential points. Literature is focusing on classical linear regressions (OLS) but a lot of insights provided by those techniques do also provide "warning" signs when using RFs and BDTs.

Have a look here: tools include

• Cook's distance - for measuring influence
• Studentized residuals - for measuring outlierness
• Leverage - for measuring "unusualness" of x's

Edit 1

I do not think that Cook's distance nor Leverage generalizes to regression trees

I'd say that Cook's distance and Leverage generalize to a tree ensemble if we assume that the ensemble should approximate a function whose behavior could also be represented with a desired level of precision by a Generalized Linear Model. Not sure if this is the case.

What I did try was something like the studentized residual. The random forest gives a standard deviation so I can do this in principle. In practice it did not work well.

What standard deviation are you referring to? For sure you get residuals; and btw, there's a connection bw leverage and studentized residuals (wiki)

Bagging

The OP mentions both Random Forests (RF) and Boosted Decision Trees (BDT). I just would like to stress that RFs and BDTs are not the same.

Bagging, in the presence of, highly influential points, is an option:

If an “unstable predictor” is defined as a predictor for which there are highly influential points, then, in most situations, our analysis is in accordance with Breiman’s: bagging stabilizes estimators and reduces variance (source)

Answer 2

If you believe these are recording errors, why not remove them (or correct them) now that you have identified them, and retrain the Boosted Decision Tree?

If your assumption about recording errors is correct, this should improve the performance of your model out-of-sample. If your assumptions are incorrect, you will get inflated performance estimates on your test set.

Since you are still refining your model, you should not be using your test set yet. Use cross-validation instead. Only use the test set at the very end to estimate the performance of your model out-of-sample. You can, however, go through the test set and correct/remove any recording errors you can spot - just do not use your model for that.