3
$\begingroup$

I'm very confused about the relationship between "bias-variance tradeoff" and "consistent model selection". Based on my current interpretation, the ultimate goal of taking care of the "bias-variance tradeoff" is to have good out-of-sample prediction. That is to say, we want to avoid both overfitting (complex model, high variance) and underfitting (simple model, high bias) so that our estimated model can make good prediction for our target $Y$ using regressors $X$ on an entirely new dataset not used before. In this process, we don't care about what the "true model" between $Y$ and $X$ is.

On the other hand, "consistent model selection" presumes that there is a true relationship between $Y$ and $X$ denoted by $Y=X^{*'}\beta+\epsilon$ for some $\beta$, where $X^*$ is a subset of $X$. The goal is to find out this correct model (correct subset of regressors) using some criterion such as BIC, which is known to find the correct model with probability going to 1 as sample size goes to infinity. Thus it seems that the topic of consistent model selection is a much narrower territory under stronger assumptions and with a completely different ultimate goal.

The thing that confuses me is that quite often I saw the illustration of the BIC criterion as trying to balance bias (good in sample fit) and variance (model complexity). Such discussions seem to imply that BIC is built for improving out-of-sample prediction. Personally, selecting the correct model is not quite related to having good out-of-sample prediction. A wrong model could do equally well or even better in terms of out-of-sample prediction. My question is, are these interpretations correct? If they are, then what do we actually mean when we talk about balancing bias and variance in the context of consistent model selection?

$\endgroup$
2
  • 1
    $\begingroup$ It is true that a wrong model may be better at prediction, but saying that selecting the correct model is not quite related to having good out-of-sample prediction is quite strong. The problem with the correct model is of course that we may have trouble estimating its parameters precisely. Having a wrong model that omits some relevant regressors produces unbounded risk (IIRC), motivating selection of larger-than-true models (as AIC tends to do) that have even more parameters to be estimated... So the intuition is not obvious to me. $\endgroup$ Commented Mar 11 at 17:54
  • $\begingroup$ @RichardHardy Thanks a lot, Richard! I agree. $\endgroup$ Commented Mar 12 at 2:05

1 Answer 1

3
$\begingroup$

Having good out-of-sample prediction is one goal of a model. In some applications, it is the only one or the main one. But, many times, interpretability is a major goal. If we are doing inferential statistics, then we want to be able to infer to a population, but what exactly we want to infer can vary. For example, if we are measuring risk of defaulting on a loan, then prediction is very important and interpretability much less so. But if we are (say) looking at the relation between depression, age, sex, and so on, then interpretability may be the main thing.

Consistent model selection does not, in my view, assume there is one correct model. It tries to find the best model (by some criterion) or, at least, a very good model. As George Box said "all models are wrong, but some are useful". We want useful models. And what that use is may vary. In my first example, the use is deciding whether to loan money (or what interest rate to charge). In the second, it is trying to figure out what causes, or leads to, or, at least, is associated with, depression.

In fact, a truly correct model would be useless. I think it was Jorge Luis Borges who wrote a story involving a perfect map -- but the map was, necessarily, just as big as the territory it mapped.

In practice, AIC, BIC, and all the other fit criteria are intended to find a good balance between complexity and "rightness" in the sample. While it is of theoretical interest what happens when N goes to infinity, in practice, N doesn't do that. And that's one reason why there is no general agreement on how to fit models. My view (and bear in mind that I nearly always worked in fields where interpretability was important) is that we need to use our brains and that there is no single algorithm that works all the time in all situations.

Balance bias and variance; separate signal from noise; or, as Einstein supposedly said "make everything as simple as possible, but not simpler."

$\endgroup$
4
  • 2
    $\begingroup$ Interpretability is important in loan default models, as the models have to comply with regulations. Perhaps not the main concern, but one may have to defend the model against allegations that it is biased against some group of people (especially if that shows empirically!). $\endgroup$ Commented Mar 11 at 17:57
  • $\begingroup$ @RichardHardy OK, thanks for the correction. I was under the impression that certain variables were not allowed to be entered into the model, but I am by mo means an expert on that. $\endgroup$
    – Peter Flom
    Commented Mar 11 at 18:06
  • $\begingroup$ Neither am I, but from what I have heard, regulation is a serious factor in such models. $\endgroup$ Commented Mar 11 at 18:20
  • $\begingroup$ @PeterFlom Thanks, Peter! This is helpful! $\endgroup$ Commented Mar 12 at 2:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.