22
$\begingroup$

I encountered some statisticians that never use models other than Linear Regression for prediction because they believe that "ML models" such as random forest or gradient boosting are hard to explain or "not interpretable".

In a Linear Regression, given that the set of assumptions is verified (normality of errors, homoskedasticity, no multi-collinearity), the t-tests provide a way to test the significance of variables, tests that to my knowledge are not available in the random forests or gradient boosting models.

Therefore, my question is if I want to model a dependent variable with a set of independent variables, for the sake of interpretability should I always use Linear Regression?

Improve this question
$\endgroup$
7
  • 6
    $\begingroup$ Depends on what you still consider linear. Generalized linear models and generalized additive models still work on the basis of a linear component being estimated, but can model a wide variety of relationships. $\endgroup$
    – Frans Rodenburg
    Commented Sep 12, 2019 at 4:40
  • 2
    $\begingroup$ Also depends what you mean by interpretable. Various ways of 'peering into the black box' have been proposed for machine learning models, but may or may not be appropriate for your goals. $\endgroup$
    – user20160
    Commented Sep 12, 2019 at 5:11
  • 5
    $\begingroup$ I don't quite see what inferential statistics and t tests have to do with interpretability, which IMO mainly is about coefficient estimates. $\endgroup$
    – Stephan Kolassa
    Commented Sep 12, 2019 at 6:52
  • 3
    $\begingroup$ @StephanKolassa "Interretability" can also pertain to function form. For example, the coefficient estimates produced by algorithmic fractional polynomial curve fitting in regression models (whether linear regresion, GLM, or something else), while obtaining nice fit, are almost certainly anti-intuitive: can you call to mind the array of shapes produced by models of the form $y_{i}=\beta_{0} + \beta_{1}x_{i}^{-3/5} + \beta_{2}x_{i}^{1/3} + \beta_{3}x_{i}^{3} + \varepsilon_{i}$, and therefore interpret the relationship between $y$ and $x$ implied by your coefficient estimates? $\endgroup$
    – Alexis
    Commented Sep 12, 2019 at 22:16
  • 2
    $\begingroup$ @UserX What you describe is still linear regression (i.e. is linear in the parameters). Contrast $y_{i} = \beta_{0} + \beta_{1}x_{i} + \beta_{2}x_{i}^{2} + \varepsilon_{i}$ with $y_{i} = \beta_{0} + \beta_{1}x_{i} + x_{i}^{\beta_{2}} +\varepsilon_{i}$: the former is a linear regression model, while the latter cannot be estimated using linear regression. $\endgroup$
    – Alexis
    Commented Sep 13, 2019 at 17:45

4 Answers 4

Reset to default
31
$\begingroup$

It is hard for me to believe that you heard people saying this, because it would be a dumb thing to say. It's like saying that you use only the hammer (including drilling holes and for changing the lightbulbs), because it's straightforward to use and gives predictable results.

Second, linear regression is not always "interpretable". If you have linear regression model with many polynomial terms, or just a lot of features, it would be hard to interpret. For example, say that you used the raw values of each of the 784 pixels from MNIST† as features. Would knowing that pixel 237 has weight equal to -2311.67 tell you anything about the model? For image data, looking at activation maps of the convolutional neural network would be much easier to understand.

Finally, there are models that are equally interpretable, e.g. logistic regression, decision trees, naive Bayes algorithm, and many more.

† - As noticed by @Ingolifs in the comment, and as discussed in this thread, MNIST may be not the best example, since this is a very simple dataset. For most of the realistic image datasets, logistic regression would not work and looking at the weights would not give any straightforward answers. However, if you look closer at the weights in the linked thread, then their interpretation is also not straightforward, for example weights for predicting "5" or "9" do not show any obvious pattern (see image below, copied from the other thread).

Improve this answer
$\endgroup$
4
  • 2
    $\begingroup$ This answer I think does a good job showing how clearly logistic regression on MNIST can be explained. $\endgroup$
    – Ingolifs
    Commented Sep 12, 2019 at 23:08
  • 1
    $\begingroup$ @Ingolifs agree, but this is an activation map, you could do the same for neural network. $\endgroup$
    – Tim
    Commented Sep 13, 2019 at 5:37
  • $\begingroup$ Regardless of what it's called, it gives a clear explanation of what the logistic regression is using to make its decisions in a way you don't really get for activation maps of neural networks. $\endgroup$
    – Ingolifs
    Commented Sep 13, 2019 at 9:40
  • 1
    $\begingroup$ @Ingolifs MNIST is maybe not the best example because it's very simple, but the point is that you'd use same method for neural network. $\endgroup$
    – Tim
    Commented Sep 13, 2019 at 11:59
13
$\begingroup$

Decision Tree would be another choice. Or Lasso Regression to create a sparse system.

Check this figure from An Introduction to Statistical Learning book. enter image description here http://www.sr-sv.com/wp-content/uploads/2015/09/STAT01.png

Improve this answer
$\endgroup$
2
  • $\begingroup$ What is the "ISL" book? $\endgroup$
    – Chris
    Commented Sep 13, 2019 at 16:00
  • 1
    $\begingroup$ @donlan amazon.com/… thanks for suggestion answer revised. $\endgroup$
    – Haitao Du
    Commented Sep 13, 2019 at 17:13
7
$\begingroup$

I would agrre with Tim's and mkt's answers - ML models are not necessarily uninterpretable. I would direct you to the Descriptive mAchine Learning EXplanations, DALEX R package, which is devoted to making ML models interpretable.

Improve this answer
$\endgroup$
2
  • 1
    $\begingroup$ the DALEX Package is indeed very interesting, do you know if something similar exists for Python? $\endgroup$
    – Victor
    Commented Sep 15, 2019 at 4:36
  • $\begingroup$ @Victor I don't know of a Python version of DALEX, but you could try calling R from Python using rpy2.readthedocs.io/en/version_2.8.x/introduction.html for example. $\endgroup$
    – babelproofreader
    Commented Sep 16, 2019 at 9:53
6
$\begingroup$

No, that is needlessly restrictive. There are a large range of interpretable models including not just (as Frans Rodenburg says) linear models, generalized linear models and generalized additive models, but also machine learning methods used for regression. I include random forests, gradient boosted machines, neural networks, and more. Just because you don't get coefficients out of machine learning models that are similar to those from linear regressions does not mean that their workings cannot be understood. It just takes a bit more work.

To understand why, I'd recommend reading this question: Obtaining knowledge from a random forest . What it shows is how you can approach making almost any machine learning model interpretable.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.