12
$\begingroup$

I found many articles that state that boosting methods are sensitive to outliers, but no article explaining why.

In my experience outliers are bad for any machine learning algorithm, but why are boosting methods singled out as particularly sensitive?

How would the following algorithms to rank in terms of sensitivity to outliers: boost-tree, random forest, neural network, SVM, and simple regression methods such as logistic regression?

$\endgroup$
  • 1
    $\begingroup$ I've edited to try to clarify (also if you put spaces at the beginning of a line, stackexchange will treat it as code). To your second para, boosting is so what? You might have to define sensitivity. $\endgroup$ – Jeremy Miles Mar 3 '15 at 23:28
  • 1
    $\begingroup$ Also, outliers and noice are not the same thing. $\endgroup$ – Jeremy Miles Mar 3 '15 at 23:28
  • $\begingroup$ I wouldn't mark this question as resolved yet. It is not clear if boosting actually suffers from outliers more than other methods or not. It seems the accepted answer was accepted mostly because of confirmation bias. $\endgroup$ – rinspy Aug 17 '17 at 15:42
  • $\begingroup$ Can you share some of these articles, please? $\endgroup$ – acnalb Dec 1 '18 at 22:34
11
$\begingroup$

Outliers can be bad for boosting because boosting builds each tree on previous trees' residuals/errors. Outliers will have much larger residuals than non-outliers, so gradient boosting will focus a disproportionate amount of its attention on those points.

$\endgroup$
  • 2
    $\begingroup$ It will be better if you can give more mathematical details to the OP! $\endgroup$ – Metariat Jul 8 '16 at 8:01
  • 5
    $\begingroup$ @Matemattica I disagree that adding mathematical details will provide additional clarity here. It would just be a symbol for tree gradients, and a learning rate subsequent trees. $\endgroup$ – Ryan Zotti Jul 8 '16 at 13:45
  • 1
    $\begingroup$ @RyanZotti: I agree with Metariat. More formal notation would resolve some confusion. For example in the sentence 'Outliers will have much larger residuals than non-outliers' you mean the residuals wrt to what? The estimated model or the true one? If the former, it is not true in general and if the latter, it is irrelevant. $\endgroup$ – user603 Oct 11 '16 at 7:32
1
$\begingroup$

The algorithms you specified are for classification, so I'm assuming you don't mean outliers in the target variable, but input variable outliers. Boosted Tree methods should be fairly robust to outliers in the input features since the base learners are tree splits. For example, if the split is x > 3 then 5 and 5,000,000 are treated the same. This may or may not be a good thing, but that's a different question.

If instead you were talking about regression and outliers in the target variable, then sensitivity of boosted tree methods would depend on the cost function used. Of course, squared error is sensitive to outliers because the difference is squared and that will highly influence the next tree since boosting attempts to fit the (gradient of the) loss. However, there are more robust error functions that can be used for boosted tree methods like Huber loss and Absolute Loss.

$\endgroup$
0
$\begingroup$

In boosting we try to pick the dataset on which the algorithm results were poor instead of randomly choosing the subset of data. These hard examples are important ones to learn, so if the data set has a lot of outliers and algorithm is not performing good on those ones than to learn those hard examples algorithm will try to pick subsets with those examples.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.