During my university days, they took great care to go through everything we could do wrong when using simple regression models. Reverse causality, omitted variable bias, heteroskedasticity, normality assumptions, non-exogenous independent variables, etc.

I'm wondering about the extent to which we should worry about these things when using other families of models, particularly tree-based methods (xgboost / random forest), and neural networks. Are there any pitfalls of simple regression models that don't matter much when using trees / neural nets?

  • 1
    $\begingroup$ All the "pitfalls" you mention are problems that apply if you want to do inference. If you are interested in predictions, I don't think any of these are a problem. $\endgroup$ – Simon Boge Brant Aug 23 at 7:24
  • 1
    $\begingroup$ Sure, there are some "pitfalls" of linear regression + Ordinary Least Squares which do not matter much in machine learning. For example, random forests do not require normality of predictors or residuals. In fact, categorical predictors are encouraged. There are some other pitfalls that do matter even in advanced universes. Example: model misspecification. If the true model is a support vector machine (SVM) with Laplace kernel and you have used an SVM with non-homogeneous polynomial kernel for estimation, the classification performance may be rather poor. $\endgroup$ – stans Aug 23 at 7:38
  • 2
    $\begingroup$ With the exception of distributional assumptions (and you don't need to assume Normal errors for linear regression, and by definition you're not assuming that for Logistic regression), I think you have to worry to some extent about all of these. There are technical assumptions that are required in order to assert that a parameter estimate in a model follows a t (linear regression) or z (logistic regression) distribution, and hence that the confidence intervals / p-values are "correct". But there are also assumptions about the data that are going to be an issue whatever method you use. $\endgroup$ – Paul Hewson Aug 23 at 8:16
  • 1
    $\begingroup$ Nonparametric/tree-based methods are more robust, but they are not a bulletproof vest. $\endgroup$ – Digio Aug 23 at 8:28
  • $\begingroup$ @stans "For example, random forests do not require normality of predictors" - neither does regression. At all. In any sense. And if you're not doing inference (tests, CIs, or PIs) that assumes normality of errors, you don't necessarily need it for anything in regression. $\endgroup$ – Glen_b Aug 23 at 9:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.