2
$\begingroup$

Tree based models such as (gradient boosting or random forest) have a lot of advantages, such as robust to collinearity and outliers.

I can see deep neural networks (MLP) are robust to collinearity. But are they robust to outliers? and Why?

$\endgroup$
  • $\begingroup$ What does "robust to collinearity" and "robust to outliers" mean? $\endgroup$ – Matthew Drury Jun 24 '17 at 15:59
2
$\begingroup$

Based on this article by Lin and Tegmark (below), I think the answer is "it depends." In other words and as they state, most deep learning algorithms assume a lognormal distribution. As long as your data fits that assumption, then there is no problem. The problem is when the tails of the distribution are more extreme than lognormal, e.g., power lawed or super-exponential. Their Figure 1 and related discussion outline the issues with the lack of tail fit based on lognormality across several different data types and solutions are proposed, specifically in the context of deep learning NNs.

Lin and Tegmark, Critical Behavior from Deep Dynamics: A Hidden Dimension in Natural Language arXiv:1606.06737

$\endgroup$
2
$\begingroup$

Relative to a standard multiple regression model, I believe an MLP is much more robust to outliers. This is the case for several reasons:

1) the multiple regression has only one single shot at fitting the data. Meanwhile the MLP has so many more opportunities to fit the data by varying the number of nodes and hidden layers to use to fit the data. This more flexible fitting mechanism should allow the MLP to underweight the impact of outliers (relative to either a Y or a X variable);

2) MLPs activation functions typically use a Logit Regression mechanism (Sigmoid) or a Tangent Hyperbolic function (Tanh). The former generates intermediary outputs between 0 and 1 and the latter between -1 and +1. Those activation functions further enhance the capability of MLPs to deal with non-linear events and outliers.

3) MLPs can incorporate regularization mechanisms. The latter should assist in resolving multicollinearity and reducing the impact of outliers.

Also, to diagnose the impact of outliers on your MLPs, you can also do cross validation.

However, if your main objective is to reduce the impact of outliers there are more transparent ways to deal with that. Tree based models are certainly a good way to do that, as you mentioned. But, there is also a whole family of Robust Regression models. Some of them combine with regularization mechanisms to resolve the multicollinearity issue. And, those models are far easier to explain to a non-specialized audience.

$\endgroup$
  • 1
    $\begingroup$ For (1) the higher complexity of a DNN actually causes it to overweight (overfit) outliers, which may be undesirable. Compared to a deep architecture, A shallow NN (e.g., a multiple regression) is less influenced by noise (outliers). $\endgroup$ – user0 Jun 24 '17 at 10:32
  • 1
    $\begingroup$ Influence can have two different meanings. Influence on the estimate, in this case you may be right. I agree that DNN tend to overfit. I am not sure overfitting is exactly the same issue as being vulnerable to outliers. Influence on the coefficient is another meaning. And, in such case I would advance that regressions are much more vulnerable to outliers than DNN. A single outlier out of 50 observations can change the statistical significance of a regression coefficient or even its sign. I think it is much less the case for DNNs. $\endgroup$ – Sympa Jun 25 '17 at 21:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.