# Statistical Learning. Contradictions?

Currently I am re-reading some chapters of: An Introduction to Statistical Learning with Applications in R by Gareth James, Daniela Witten, Trevor Hastie and Robert Tibshirani (Springer, 2015). Now, I have some doubts about what is said there.

Above all it seems to me relevant to note that in chapter 2 two concepts are introduced : prediction accuracy-model interpretability tradeoff and bias-variance tradeoff. I mentioned the latter in an earlier question.

In this book, it is suggested that focusing on expected prediction error (test MSE) yields the following assertions:

• less flexible specifications imply more bias but less variance

• more flexible specifications imply less bias but more variance

It follows that linear regression implies more bias but less variance. The optimum in the tradeoff between bias and variance, the minimum in test MSE, depends on the true form of $$f()$$ [in $$Y = f(X) + \epsilon$$]. Sometimes linear regression works better than more flexible alternatives and sometimes not. This graph tells this story:

In the second case linear regression works quite well, in the others two not so much. All is ok in this perspective.

In my opinion the problem appears under the perspective of inference and interpretability used in this book. In fact this book also suggests that:

• less flexible specifications are more far away from reality, then more biased, but at the same time they are more tractable and, then, more interpretable;

• more flexible specifications are closer to the reality, hence less biased, but at the same time they are less tractable and, then, less interpretable.

As a result we have that linear regressions, OLS and even more LASSO, are the most interpretable and more powerful for inference. This graph tell this story:

This seems to me like a contradiction. How is possible that linear models are, at the same time, the more biased but the best for inference? And among linear models, how is possible that LASSO regression is better than OLS one for inference?

EDIT: My question can summarized as:

• linear estimated model are indicated as the more interpretable even if the more biased.

• linear estimated model are indicated as the more reliable for inference even if the more biased.

I read carefully the answer and comments of Tim. However it seems to me that some problems remain. So, actually it looks like in some sense the first condition can hold, i.e. in a sense where “interpretability” is a property of the estimated model itself (its relation with something "outside" are not considered).

About inference "outside" is the core, but the problem can move around its precise meaning. Then, I checked the definition that Tim suggested (What is the definition of Inference?), also here (https://en.wikipedia.org/wiki/Statistical_inference), and elsewhere. Some definition are quite general but in most material that I have inference is intended as something like: from sample say something about the "true model", regardless of his deep meaning. So, the Authors of the book under consideration used something like the “true model”, implying we cannot skip it. Now, any biased estimator cannot say something right about the true model and/or its parameters, even asymptotically. Unbiasedness/consistency (difference irrelevant here) is the main requirements for any model written for pure inference goal. Therefore the second condition cannot hold, and the contradiction remains.

• No model in statistics/ML estimates the "true model". Our models always approximate the true model, bias and variance are ways of measuring the problems with this approximation. – Tim Oct 27 '20 at 9:01
• I agree. However inference move around the true model not its approximation. If the model is severily missspecified (bad approximated) inference from it cannot be better than one that come from a better specified one. – markowitz Oct 27 '20 at 10:10
• Yes, but more biased (underfitting) model is not any worse (it it isn't better) for inference, than more varying (overfitting). – Tim Oct 27 '20 at 10:26
• In my view under/overfitting are concept useful for prediction more than inference. Between prediction and inference (=explanation about real world) something can be shared, but these area must be clearly separated. This point is precisely addressed in this article (stat.berkeley.edu/~aldous/157/Papers/…) that I have already mentioned in comments. – markowitz Oct 27 '20 at 13:13
• In it is also shown as more biased/misspecified model can perform better than correct one (or less biased) if the prediction is the goal. This is the heart of bias-variance tradeoff, and as I underscored elsewhere this point, unfortunately, is forget sometimes. However this tradeoff don't play important role if the goal is pure explanation/inference; in it bias minimization is the core. Therefore say that among several alternatives the more biased model is better for inference … is a wrong message. This is my main point. – markowitz Oct 27 '20 at 13:13