Paradox in model selection (AIC, BIC, to explain or to predict?)

Question

Having read Galit Shmueli's "To Explain or to Predict" (2010) and some literature on model selection using AIC and BIC, I am puzzled by an apparent contradiction. There are three premises,

1. AIC- versus BIC-based model choice (end of p. 300 - start of p. 301): simply put, AIC should be used for selecting a model intended for prediction while BIC should be used for selecting a model for explanation. Additionally (not in the above paper), we know that under some conditions BIC selects the true model among the set of candidate models; the true model is what we seek in explanatory modelling (end of p. 293).
2. Simple arithmetics: AIC will select a larger model than BIC for samples of size 8 or larger (satisfying $\text{ln}(n)>2$ due to the different complexity penalties in AIC versus BIC).
3. The true model (i.e. the model with the correct regressors and the correct functional form but imperfectly estimated coefficients) may not be the best model for prediction (p. 307): a regression model with a missing predictor may be a better forecasting model -- the introduction of bias due to the missing predictor may be outweighted by the reduction in variance due to estimation imprecision.

Points 1. and 2. suggest that larger-than-true models may be better for prediction than more parsimonious models. Meanwhile, point 3. gives an opposite example where a more parsimonious model is better for prediction than a larger, true model. I find this puzzling.

Questions:

1. How can the apparent contradiction between points {1. and 2.} and 3. be explained/resolved?
2. In light of point 3., could you give an intuitive explanation for why and how a larger model selected by AIC is actually better for prediction than a more parsimonious model selected by BIC?

---

_{I am not saying there is a contradiction in Shmueli (2010), I am just trying to understand an apparent paradox.}

Answer 1

I will try to explain what's going on with some materials that I am referring to and what I have learned with personal correspondence with the author of the materials.

Above is an example where we are trying to infer a 3rd degree polynomial plus noise. If you look at the bottom left quadrant, you will see that on a cumulative basis AIC beats BIC on a 1000 sample horizon. However you can also see that up to sample 100, instantaneous risk of AIC is worse that BIC. This is due to the fact that AIC is a bad estimator for small samples (a suggested fix is AICc). 0-100 is the region where "To Explain or To Predict" paper is demonstrating without a clear explanation of what's going on. Also even though it is not clear from the picture when the number of samples become large (the slopes become almost identical) BIC instantaneous risk outperforms AIC because the true model is in the search space. However at this point the ML estimates are so much concentrated around their true values that the overfitting of AIC becomes irrelevant as the extra model parameters are very very close to 0. So as you can see from the top-right quadrant AIC identifies on average a polynomial degree of ~3.2 (over many simulation runs it sometimes identifies a degree of 3 sometimes 4). However that extra parameter is minuscule, which makes AIC a no-brainer against BIC.

The story is not that simple however. There are several confusions in papers treating AIC and BIC. Two scenarios to be considered:

1) The model that is searched for is static/fixed, and we increase the number of samples and see what happens under different methodologies.

a) The true model is in search space. We covered this case above.

b) The true model is not in search space but can be approximated with the functional form we are using. In this case AIC is also superior.

http://homepages.cwi.nl/~pdg/presentations/RSShandout.pdf (page 9)

c) The true model is not in search space and we are not even close to getting in right with an approximation. According to Prof. Grunwald, we don't know what's going on under this scenario.

2) The number of samples are fixed, and we vary the model to be searched for to understand the effects of model difficulty under different methodologies.

Prof. Grunwald provides the following example. The truth is say a distribution with a parameter $\theta = \sqrt{(\log n) / n}$ where n is the sample size. And the candidate model 1 is $\theta = 0$ and candidate model 2 is a distribution with a free parameter $\theta^*$. BIC always selects model 1, however model 2 always predicts better because the ML estimate is closer to $\theta$ than 0. As you can see BIC is not finding the truth and and also predicting worse at the same time.

There is also the non-parametric case, but I don't have much information on that front.

My personal opinion is that all the information criteria are approximations and one should not expect a correct result in all cases. I also believe that the model that predicts best is also the model that explains best. It is because when people use the term "model" they don't involve the values of the parameters just the number the parameters. But if you think of it as a point hypothesis then the information content of the protested extra parameters are virtually zero. That's why I would always choose AIC over BIC, if I am left with only those options.

Answer 2

They are not to be taken in the same context; points 1 and 2 have different contexts. For both AIC and BIC one first explores which combination of parameters in which number yield the best indices (Some authors have epileptic fits when I use the word index in this context. Ignore them, or look up index in the dictionary.) In point 2, AIC is the richer model, where richer means selecting models with more parameters, only sometimes, because frequently the optimum AIC model is the same number of parameters model as BIC the selection. That is, if AIC and BIC select models having the SAME number of parameters then the claim is that AIC will be better for prediction than BIC. However, the opposite could occur if BIC maxes out with a fewer parameters model selected (but no guarantees). Sober (2002) concluded that AIC measures predictive accuracy while BIC measures goodness of fit, where predictive accuracy can mean predicting y outside of the extreme value range of x. When outside, frequently a less optimal AIC having weakly predictive parameters dropped will better predict extrapolated values than an optimal AIC index from more parameters in its selected model. I note in passing that AIC and ML do not obviate the need for extrapolation error testing, which is a separate test for models. This can be done by withholding extreme values from the "training" set and computing the error between the extrapolated "post-training" model and the withheld data.

Now BIC is supposedly a lesser error predictor of y-values within the extreme values of range of x. Improved goodness of fit often comes at the price of bias of the regression (for extrapolation), wherein the error is reduced by introducing that bias. This will, for example, often flatten the slope to split the sign of the average left verses right $f(x)-y$ residuals (think of more negative residuals on one side and more positive residuals on the other) thereby reducing total error. So in this case we are asking for the best y value given an x value, and for AIC we are more closely asking for a best functional relationship between x and y. One difference between these is, for example, that BIC, other parameter choices being equal, will have a better correlation coefficient between model and data, and AIC will have better extrapolation error measured as y-value error for a given extrapolated x-value.

Point 3 is a sometimes statement under some conditions

• when the data are very noisy (large $σ$);

• when the true absolute values of the left-out parameters (in our
  example $β_2$) are small;

• when the predictors are highly correlated; and

• when the sample size is small or the range of left-out variables is small.

In practice, a correct form of an equation does not mean that fitting with it will yield the correct parameter values because of noise, and the more noise the merrier. The same thing happens with R$^2$ versus adjusted R$^2$ and high collinearity. That is, sometimes when a parameter is added adjusted R$^2$ degrades while R$^2$ improves.

I would hasten to point out that these statements are optimistic. Typically, models are wrong, and often a better model will enforce a norm that cannot be used with AIC or BIC, or the wrong residual structure is assumed for their application, and alternative measures are needed. In my work, this is always the case.

Answer 3

I read Shmueli's "To Explain or to Predict" (2010) a couple of years ago for the first time and it was one of the most important readings for me. Several great doubts come to solve after such reading.

It seems me that the contradictions you notice are less relevant that it seems to be. I try to reply to your two questions together.

My main argument is that your point 3 do not appear at pag 307 (here there are the detail) but at the begin of the discussion – bias-variance tradeoff argument (par 1.5; in particular end of pag 293). Your point 3 is the core message of the article. (See EDIT)

Your points 1 and 2 are related to the sub-argument of model selection. At this stage the main important practical difference between explanatory and predictive models do not appear. The analysis of the predictive models must involve out of sample data, in explanatory models it is not the case.

In predictive framework, firstly we have model estimation, then model selection that is something like evaluate the model (hyper)parameters tuning; at the end we have model evaluation on new data.

In explanatory framework, model estimation/selection/evaluation are much less distinguishable. In this framework theorethical consideration seems me much more important that the detailed distinction between BIC and AIC.

In Shmueli (2010) the concept of true model is intended as theoretical summary that imply substantial causal meaning. Causal inference is the goal. [For example you can read: “proper explanatory model selection is performed in a constrained manner … A researcher might choose to retain a causal covariate which has a strong theoretical justification even if is statistically insignificant.” Pag 300]

Now, the role of true model in causal inference debate is of my great interest and represent the core of several question that I opened on this web-community. For example you can read:

Regression and causality in econometrics

Structural equation and causal model in economics

Causality: Structural Causal Model and DAG

Today my guess is that the usual concept of true model is too simplistic for carried out exhaustive causal inference. At the best we can interpret it as very particular type of Pearl’s Structural Causal Model.

I know that, under some condition, BIC method permit us to select the true model. However the story that is behind this result sound me as too poor for exhaustive causal inference.

Finally the distinction between AIC and BIC seems me not so important and, most important, it does not affect the main point of the article (your 3).

EDIT: To be clearer. The main message of the article is that explanation and prediction are different things. Prediction and explanation (causation) are different goal that involve different tools. Conflation between them without understood the difference is a big problem. Bias-variance tradeoff is the main theoretical point that justify the necessity of the distinction between prediction and explanation. In this sense your point 3 is the core of the article.

EDIT2 In my opinion the fact here is that the problems addressed by this article are too wide and complex. Then, more than as usual, concepts like contradiction and/or paradox should be contextualized. For some readers that reads your question but not the article can seems that the article at all, or at least in most part, should be refuse, until somebody do not resolve the contradiction. My point is that this is not the case.

Suffice to say that the author could simply skip model selection details and the core message could remain the same, definitely. In fact the core of the article is not about the best strategy to achieve good prediction (or explanation) model, but to show that prediction and explanation are different goal that imply different method. In this sense your point 1 and 2 are minor and this fact resolve the contradiction (in the sense above).

At the other side remain the fact that AIC bring us to prefer long rather then short regression and this fact contradicts the argument at your point 3 is refer to. In this sense the paradox and or contradiction remain.

Maybe the paradox come from the fact that the argument behind point 3, bias-variance trade-off, is valid in finite sample data; in small sample can be substantial. In case of infinitely large sample, estimation error of parameter disappear, but possible bias term no, then the true model (in empirical sense) become the best also in the sense of expected prediction error. Now the good prediction properties of AIC is achieved only asymptotically, in small sample it can select models that have too many parameters then overfitting can appear. In case like this is hard to say precisely in what way the sample size matters.

However in order to face the problem of small sample a modified version of AIC was developed. See here: https://en.wikipedia.org/wiki/Akaike_information_criterion#Modification_for_small_sample_size

I done some calculus as examples and if these are free of mistake:

for the case of 2 parameters (as the case in Shmueli example) if we have less than 8 obs AIC penalizes more than BIC (as you says). If we have more than 8 but less than 14 obs AICc penalizes more than BIC. If we have 14 or more obs BIC is again the more penalizer

for the case of 5 parameters, if we have less than 8 obs AIC penalizes more than BIC (as you says). If we have more than 8 but less than 19 obs AICc penalizes more than BIC. If we have 19 or more obs BIC is again the more penalizer

for the case of 10 parameters, if we have less than 8 obs AIC penalizes more than BIC (as you says). If we have more than 8 but less than 28 obs AICc penalizes more than BIC. If we have 28 or more obs BIC is again the more penalizer.

Finally let me remark that if we remain very close to author words we can read that she do not explicitly suggest to use AIC in prediction and BIC in explanation (as reported at your point 1). She essentially said that: in explanatory model theoretical consideration are relevant and in prediction no. This is the core of the difference between these two kind of model selection. Then AIC is just presented as “popular metric” and its popularity come from the idea behind it. We can read: “A popular predictive metric is the in-sample Akaike Information Criterion (AIC). Akaike derived the AIC from a predictive viewpoint, where the model is not intended to accurately infer the “true distribution,” but rather to predict future data as accurately as possible”.