15
$\begingroup$

I found many articles that state that boosting methods are sensitive to outliers, but no article explaining why.

In my experience outliers are bad for any machine learning algorithm, but why are boosting methods singled out as particularly sensitive?

How would the following algorithms to rank in terms of sensitivity to outliers: boost-tree, random forest, neural network, SVM, and simple regression methods such as logistic regression?

$\endgroup$
4
  • 1
    $\begingroup$ I've edited to try to clarify (also if you put spaces at the beginning of a line, stackexchange will treat it as code). To your second para, boosting is so what? You might have to define sensitivity. $\endgroup$ Mar 3 '15 at 23:28
  • 1
    $\begingroup$ Also, outliers and noice are not the same thing. $\endgroup$ Mar 3 '15 at 23:28
  • $\begingroup$ I wouldn't mark this question as resolved yet. It is not clear if boosting actually suffers from outliers more than other methods or not. It seems the accepted answer was accepted mostly because of confirmation bias. $\endgroup$
    – rinspy
    Aug 17 '17 at 15:42
  • $\begingroup$ Can you share some of these articles, please? $\endgroup$
    – acnalb
    Dec 1 '18 at 22:34
18
$\begingroup$

Outliers can be bad for boosting because boosting builds each tree on previous trees' residuals/errors. Outliers will have much larger residuals than non-outliers, so gradient boosting will focus a disproportionate amount of its attention on those points.

$\endgroup$
3
  • 2
    $\begingroup$ It will be better if you can give more mathematical details to the OP! $\endgroup$
    – Metariat
    Jul 8 '16 at 8:01
  • 5
    $\begingroup$ @Matemattica I disagree that adding mathematical details will provide additional clarity here. It would just be a symbol for tree gradients, and a learning rate subsequent trees. $\endgroup$
    – Ryan Zotti
    Jul 8 '16 at 13:45
  • 2
    $\begingroup$ @RyanZotti: I agree with Metariat. More formal notation would resolve some confusion. For example in the sentence 'Outliers will have much larger residuals than non-outliers' you mean the residuals wrt to what? The estimated model or the true one? If the former, it is not true in general and if the latter, it is irrelevant. $\endgroup$
    – user603
    Oct 11 '16 at 7:32
3
$\begingroup$

The algorithms you specified are for classification, so I'm assuming you don't mean outliers in the target variable, but input variable outliers. Boosted Tree methods should be fairly robust to outliers in the input features since the base learners are tree splits. For example, if the split is x > 3 then 5 and 5,000,000 are treated the same. This may or may not be a good thing, but that's a different question.

If instead you were talking about regression and outliers in the target variable, then sensitivity of boosted tree methods would depend on the cost function used. Of course, squared error is sensitive to outliers because the difference is squared and that will highly influence the next tree since boosting attempts to fit the (gradient of the) loss. However, there are more robust error functions that can be used for boosted tree methods like Huber loss and Absolute Loss.

$\endgroup$
0
1
$\begingroup$

A nice literature review of this topic can be found in

Alexander Hanbo Li and Jelena Bradic "Boosting in the presence of outliers: adaptive classification with non-convex loss functions" Journal of the American Statistical Association. 2018. Preprint link: https://arxiv.org/abs/1510.01064

Recent advances in technologies for cheaper and faster data acquisition and storage have led to an explosive growth of data complexity in a variety of research areas such as high-throughput genomics, biomedical imaging, high-energy physics, astronomy and economics. As a result, noise accumulation, experimental variation and data inhomogeneity have become substantial. Therefore, developing classification methods that are highly efficient and accurate in such settings, is a problem that is of great practical importance. However, classification in such settings is known to poses many statistical challenges and calls for new methods and theories. For binary classification problems, we assume the presence of separable, noiseless data that belong to two classes and in which an adversary has corrupted a number of observations from both classes independently. There are a number of setups that belong to this general framework. A random flipped label design, in which the labels of the class membership were randomly flipped, is one example that can occur very frequently, as labeling is prone to a number of errors, human or otherwise. Another example is the presence of outliers in the observations, in which a small number of observations from both classes have a variance that is larger than the noise of the rest of the observations. Such situations may naturally occur with the new era of big and heterogeneous data, in which data are corrupted (arbitrarily or maliciously) and subgroups may behave differently; a subgroup might only be one or a few individuals in small studies that would appear to be outliers within class data.

Considerable effort has therefore been focused on finding methods that adapt to the relative error in the data. Although this has resulted in algorithms, e.g. Grünwald and Dawid (2004), that achieve provable guarantees (Natarajan et al., 2013; Kanamori et.al, 2007) when contamination model (Scott et al., 2013) is known or when multiple noisy copies of the data are available (Cesa-Bianchi et al., 2011), good generalization errors in the test set are by no means guaranteed. This problem is compounded when the contamination model is unknown, where outliers need to be detected automatically. Despite progress on outlier-removing algorithms, significant practical challenges (due to exceedingly restrictive conditions imposed therein) remain. In this paper, we concentrate on the ensemble algorithms. Among these, AdaBoost (Freund and Schapire, 1997) has proven to be simple and effective in solving classification problems of many different kinds. The aesthetics and simplicity of AdaBoost and other forward greedy algorithms, such as LogitBoost (Friedman, et al., 2000), also facilitated a tacit defense from overfitting, especially when combined with early termination of the algorithm (Zhang and Yu, 2005). Friedman, et al. (2000) developed a powerful statistical perspective, which views AdaBoost as a gradient-based incremental search for a good additive model using the exponential loss. The gradient boosting (Friedman, 2001) and AnyBoost (Mason et al., 1999) have used this approach to generalize the boosting idea to wider families of problems and loss functions. This criterion was motivated by the fact that the exponential loss is a convex surrogate of the hinge or 0 − 1 loss. Nevertheless, in the presence of label noise and/or outliers, the performance of all of them deteriorates rapidly (Dietterich, 2000). Although algorithms like LogitBoost, MadaBoost (Domingo and Watanabe, 2000), Log-lossBoost (Collins, et al., 2002) are able to better tolerate noise than AdaBoost, they are still not insensitive to outliers. Hence, they are efficient when the data is observed with little or no noise. However, Long and Servedio (2010) pointed out that any boosting algorithm with convex loss functions is highly susceptible to a random label noise model. They constructed a simple example, from hereon denoted Long/Servedio problem, that cannot be “learned” by the boosting algorithms above.

$\endgroup$
-1
$\begingroup$

In boosting we try to pick the dataset on which the algorithm results were poor instead of randomly choosing the subset of data. These hard examples are important ones to learn, so if the data set has a lot of outliers and algorithm is not performing good on those ones than to learn those hard examples algorithm will try to pick subsets with those examples.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.