7
$\begingroup$

Currently I am re-reading some chapters of: An Introduction to Statistical Learning with Applications in R by Gareth James, Daniela Witten, Trevor Hastie and Robert Tibshirani (Springer, 2015). Now, I have some doubts about what is said there.

Above all it seems to me relevant to note that in chapter 2 two concepts are introduced : prediction accuracy-model interpretability tradeoff and bias-variance tradeoff. I mentioned the latter in an earlier question.

In this book, it is suggested that focusing on expected prediction error (test MSE) yields the following assertions:

  • less flexible specifications imply more bias but less variance

  • more flexible specifications imply less bias but more variance

It follows that linear regression implies more bias but less variance. The optimum in the tradeoff between bias and variance, the minimum in test MSE, depends on the true form of $f()$ [in $Y = f(X) + \epsilon$]. Sometimes linear regression works better than more flexible alternatives and sometimes not. This graph tells this story:

enter image description here

In the second case linear regression works quite well, in the others two not so much. All is ok in this perspective.

In my opinion the problem appears under the perspective of inference and interpretability used in this book. In fact this book also suggests that:

  • less flexible specifications are more far away from reality, then more biased, but at the same time they are more tractable and, then, more interpretable;

  • more flexible specifications are closer to the reality, hence less biased, but at the same time they are less tractable and, then, less interpretable.

As a result we have that linear regressions, OLS and even more LASSO, are the most interpretable and more powerful for inference. This graph tell this story:

enter image description here

This seems to me like a contradiction. How is possible that linear models are, at the same time, the more biased but the best for inference? And among linear models, how is possible that LASSO regression is better than OLS one for inference?

EDIT: My question can summarized as:

  • linear estimated model are indicated as the more interpretable even if the more biased.

  • linear estimated model are indicated as the more reliable for inference even if the more biased.

I read carefully the answer and comments of Tim. However it seems to me that some problems remain. So, actually it looks like in some sense the first condition can hold, i.e. in a sense where “interpretability” is a property of the estimated model itself (its relation with something "outside" are not considered).

About inference "outside" is the core, but the problem can move around its precise meaning. Then, I checked the definition that Tim suggested (What is the definition of Inference?), also here (https://en.wikipedia.org/wiki/Statistical_inference), and elsewhere. Some definition are quite general but in most material that I have inference is intended as something like: from sample say something about the "true model", regardless of his deep meaning. So, the Authors of the book under consideration used something like the “true model”, implying we cannot skip it. Now, any biased estimator cannot say something right about the true model and/or its parameters, even asymptotically. Unbiasedness/consistency (difference irrelevant here) is the main requirements for any model written for pure inference goal. Therefore the second condition cannot hold, and the contradiction remains.

$\endgroup$
  • 4
    $\begingroup$ No model in statistics/ML estimates the "true model". Our models always approximate the true model, bias and variance are ways of measuring the problems with this approximation. $\endgroup$ – Tim Oct 27 at 9:01
  • $\begingroup$ I agree. However inference move around the true model not its approximation. If the model is severily missspecified (bad approximated) inference from it cannot be better than one that come from a better specified one. $\endgroup$ – markowitz Oct 27 at 10:10
  • $\begingroup$ Yes, but more biased (underfitting) model is not any worse (it it isn't better) for inference, than more varying (overfitting). $\endgroup$ – Tim Oct 27 at 10:26
  • $\begingroup$ In my view under/overfitting are concept useful for prediction more than inference. Between prediction and inference (=explanation about real world) something can be shared, but these area must be clearly separated. This point is precisely addressed in this article (stat.berkeley.edu/~aldous/157/Papers/…) that I have already mentioned in comments. $\endgroup$ – markowitz Oct 27 at 13:13
  • $\begingroup$ In it is also shown as more biased/misspecified model can perform better than correct one (or less biased) if the prediction is the goal. This is the heart of bias-variance tradeoff, and as I underscored elsewhere this point, unfortunately, is forget sometimes. However this tradeoff don't play important role if the goal is pure explanation/inference; in it bias minimization is the core. Therefore say that among several alternatives the more biased model is better for inference … is a wrong message. This is my main point. $\endgroup$ – markowitz Oct 27 at 13:13
9
$\begingroup$

There’s no contradiction. The fact that something is easy to interpret has nothing to do with how accurate is it. The most interpretable model you could imagine is to predict constant, independently of the data. In such case, you would always be able to explain why your model made the prediction it made, but the predictions would be horrible.

That said, it’s not the case that you need complicated, black-box models if you want accurate results and poorly performing models for interpretability. Here you can find nice, popular article by Cynthia Rudin and Joanna Radin, where they give example of interpretable models giving very good results and use it to discuss how performance vs interpretability is a false dichotomy. There’s also very interesting episode of Data Skeptic podcast on this subject hosting Cynthia Rudin.

You may be interested also in the When is a biased estimator preferable to unbiased one? thread.

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ Thanks for answer and links. Useful. However I’m worry about to not misunderstand what you, and the references you suggest, said. The doubts for me move around the words: interpretability, explanation, inference. You, and the suggested article of Rudin and Radin, never speak about “inference” but in my question I use it not by chance. You don’t use it by chance or because consider inference and interpretability not like synonyms? $\endgroup$ – markowitz Oct 26 at 8:14
  • $\begingroup$ You can forget for a moment the figure 2.7 and can consider that Authors of book suggest that linear model are, at the same time, the more reliable for inference and the more biased. In other words you can put something like “reliability for inference” in the horizontal axis of the figure 2.7. If you don't trust me I can report some phrase about it. Now if we use the word “interpretability” in place of something like simplicity (no more), I agree with you. In this sense the constant model that you suggest, regardless his prediction performance, is easily interpretable but bad for inference. $\endgroup$ – markowitz Oct 26 at 8:15
  • $\begingroup$ At the other side, if you affirm that constant model is useful for inference too, I disagree …. or at least we have to go deeper in the meaning of “inference”. $\endgroup$ – markowitz Oct 26 at 8:16
  • $\begingroup$ Now, in the article of Rudin and Radin word like "explanation" is used many times. In my knowledge it is a synonym of causality. So, if in the horizontal axis of the figure 2.7 we put something like “reliability for causal inference” the problems become more and deeper and would be far from solved by your example and references you cited. $\endgroup$ – markowitz Oct 26 at 8:16
  • 2
    $\begingroup$ @markowitz "explainability" has nothing to do with causality. Explainability is about our ability to explain how did some algorithm made some prediction. This is complicated for something like deep learning, since there's a lot of moving parts and simple in case of linear regression, or decision tree. $\endgroup$ – Tim Oct 26 at 8:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.