22
$\begingroup$

Having read Galit Shmueli's "To Explain or to Predict" (2010) and some literature on model selection using AIC and BIC, I am puzzled by an apparent contradiction. There are three premises,

  1. AIC- versus BIC-based model choice (end of p. 300 - start of p. 301): simply put, AIC should be used for selecting a model intended for prediction while BIC should be used for selecting a model for explanation. Additionally (not in the above paper), we know that under some conditions BIC selects the true model among the set of candidate models; the true model is what we seek in explanatory modelling (end of p. 293).
  2. Simple arithmetics: AIC will select a larger model than BIC for samples of size 8 or larger (satisfying $\text{ln}(n)>2$ due to the different complexity penalties in AIC versus BIC).
  3. The true model (i.e. the model with the correct regressors and the correct functional form but imperfectly estimated coefficients) may not be the best model for prediction (p. 307): a regression model with a missing predictor may be a better forecasting model -- the introduction of bias due to the missing predictor may be outweighted by the reduction in variance due to estimation imprecision.

Points 1. and 2. suggest that larger-than-true models may be better for prediction than more parsimonious models. Meanwhile, point 3. gives an opposite example where a more parsimonious model is better for prediction than a larger, true model. I find this puzzling.

Questions:

  1. How can the apparent contradiction between points {1. and 2.} and 3. be explained/resolved?
  2. In light of point 3., could you give an intuitive explanation for why and how a larger model selected by AIC is actually better for prediction than a more parsimonious model selected by BIC?

I am not saying there is a contradiction in Shmueli (2010), I am just trying to understand an apparent paradox.

$\endgroup$
  • 2
    $\begingroup$ I don't get the paradox/contradiction. AIC is efficient (asymptotically minimizes the expected prediction error) and BIC is consistent (asymptotically selects the true order). Point 3) says that bias may be outweighted by variance. There is obviously no guarantee that one is better than the other in a certain sample. So your "paradox" appears to be that for a given sample, AIC may not be best for prediction, which no surprise. For your Q2: if the bias increase induced by BIC's smaller model is larger than the variance increase in AIC's larger, AIC is better. $\endgroup$ – hejseb Jan 25 '16 at 6:42
  • 2
    $\begingroup$ I would suggest that you look at the first chapters in "Model Selection and Model Averaging" by Nils Hjort and Gerda Claeskens, maybe that will clear things up. $\endgroup$ – hejseb Jan 25 '16 at 6:43
  • $\begingroup$ After some elaborate answers and discussions, it seems to me that the apparent paradox arises because of incompatibility of the settings of (1. & 2.) and 3. In 3., the example only holds in sufficiently small samples. Regarding (1. & 2.), the theories behind AIC and BIC are asymptotic. Even worse, at least in one of the theories the true model is a moving target; it changes with the sample size. If we were to find a common setting, at least one of 1., 2. and 3. would not hold. This would invalidate the paradox. $\endgroup$ – Richard Hardy Feb 18 at 16:45
  • $\begingroup$ Moreover, the fact that models selected by AIC are better on average does not mean the ones among them that are larger than the true model are better than the true models. If my reasoning is correct, the remaining work is to write down the explanation carefully and in sufficient detail. $\endgroup$ – Richard Hardy Feb 18 at 16:45
1
$\begingroup$

I will try to explain what's going on with some materials that I am referring to and what I have learned with personal correspondence with the author of the materials.

http://homepages.cwi.nl/~pdg/presentations/RSShandout.pdf

Above is an example where we are trying to infer a 3rd degree polynomial plus noise. If you look at the bottom left quadrant, you will see that on a cumulative basis AIC beats BIC on a 1000 sample horizon. However you can also see that up to sample 100, instantaneous risk of AIC is worse that BIC. This is due to the fact that AIC is a bad estimator for small samples (a suggested fix is AICc). 0-100 is the region where "To Explain or To Predict" paper is demonstrating without a clear explanation of what's going on. Also even though it is not clear from the picture when the number of samples become large (the slopes become almost identical) BIC instantaneous risk outperforms AIC because the true model is in the search space. However at this point the ML estimates are so much concentrated around their true values that the overfitting of AIC becomes irrelevant as the extra model parameters are very very close to 0. So as you can see from the top-right quadrant AIC identifies on average a polynomial degree of ~3.2 (over many simulation runs it sometimes identifies a degree of 3 sometimes 4). However that extra parameter is minuscule, which makes AIC a no-brainer against BIC.

The story is not that simple however. There are several confusions in papers treating AIC and BIC. Two scenarios to be considered:

1) The model that is searched for is static/fixed, and we increase the number of samples and see what happens under different methodologies.

a) The true model is in search space. We covered this case above.

b) The true model is not in search space but can be approximated with the functional form we are using. In this case AIC is also superior.

http://homepages.cwi.nl/~pdg/presentations/RSShandout.pdf (page 9)

c) The true model is not in search space and we are not even close to getting in right with an approximation. According to Prof. Grunwald, we don't know what's going on under this scenario.

2) The number of samples are fixed, and we vary the model to be searched for to understand the effects of model difficulty under different methodologies.

Prof. Grunwald provides the following example. The truth is say a distribution with a parameter $\theta = \sqrt{(\log n) / n}$ where n is the sample size. And the candidate model 1 is $\theta = 0$ and candidate model 2 is a distribution with a free parameter $\theta^*$. BIC always selects model 1, however model 2 always predicts better because the ML estimate is closer to $\theta$ than 0. As you can see BIC is not finding the truth and and also predicting worse at the same time.

There is also the non-parametric case, but I don't have much information on that front.

My personal opinion is that all the information criteria are approximations and one should not expect a correct result in all cases. I also believe that the model that predicts best is also the model that explains best. It is because when people use the term "model" they don't involve the values of the parameters just the number the parameters. But if you think of it as a point hypothesis then the information content of the protested extra parameters are virtually zero. That's why I would always choose AIC over BIC, if I am left with only those options.

$\endgroup$
  • $\begingroup$ You write: I also believe that the model that predicts best is also the model that explains best. I completely disagree with this sentence. Taking apart any detail of model selection criteria, your is exactly the wrong conclusion that can come from the question. Exactly for this reason I warned of this conclusion in my reply. Basically, explanatory (causal) models are focused on unbiasedness while prediction models face overfitting. The last concept involve out of sample data, the former no. $\endgroup$ – markowitz Feb 16 at 15:13
  • $\begingroup$ This difference emerge clearly in bias-variance trade-off argument that is markedly faced in any serious statistical learning book that I have ever seen until now (keep in mind predictive learning). In econometrics where both the causal and predictive query are, or should be, very relevant the bias-variance trade-off argument are ignored in many general books. I realized this weakness a few years ago. In my opinion this was a dramatic error in econometric didactic and or literature at all. $\endgroup$ – markowitz Feb 16 at 15:13
  • $\begingroup$ Some authors started take into consideration the point above and revised their manuals in the consequent directions. Stock and Watson 4th edition is a notable example. Moreover, maybe the paradigm of true model is not good at all for causal inference (see here: stats.stackexchange.com/questions/377004/… and read Angrist and Pieshe 2017) but this is another story. $\endgroup$ – markowitz Feb 16 at 15:14
  • $\begingroup$ Also, this sentence It is because when people use the term "model" they don't involve the values of the parameters just the number the parameters , is completely false in causal analysis. $\endgroup$ – markowitz Feb 16 at 15:14
  • 1
    $\begingroup$ Incautious comments like your come from too impatience, unjustified irritation and, let me says, some ignorance. If you keep calm and follow my arguments and read the references that I suggest, you can solve from yourself your missunderstanding. $\endgroup$ – markowitz Feb 16 at 22:41
1
$\begingroup$

They are not to be taken in the same context; points 1 and 2 have different contexts. For both AIC and BIC one first explores which combination of parameters in which number yield the best indices (Some authors have epileptic fits when I use the word index in this context. Ignore them, or look up index in the dictionary.) In point 2, AIC is the richer model, where richer means selecting models with more parameters, only sometimes, because frequently the optimum AIC model is the same number of parameters model as BIC the selection. That is, if AIC and BIC select models having the SAME number of parameters then the claim is that AIC will be better for prediction than BIC. However, the opposite could occur if BIC maxes out with a fewer parameters model selected (but no guarantees). Sober (2002) concluded that AIC measures predictive accuracy while BIC measures goodness of fit, where predictive accuracy can mean predicting y outside of the extreme value range of x. When outside, frequently a less optimal AIC having weakly predictive parameters dropped will better predict extrapolated values than an optimal AIC index from more parameters in its selected model. I note in passing that AIC and ML do not obviate the need for extrapolation error testing, which is a separate test for models. This can be done by withholding extreme values from the "training" set and computing the error between the extrapolated "post-training" model and the withheld data.

Now BIC is supposedly a lesser error predictor of y-values within the extreme values of range of x. Improved goodness of fit often comes at the price of bias of the regression (for extrapolation), wherein the error is reduced by introducing that bias. This will, for example, often flatten the slope to split the sign of the average left verses right $f(x)-y$ residuals (think of more negative residuals on one side and more positive residuals on the other) thereby reducing total error. So in this case we are asking for the best y value given an x value, and for AIC we are more closely asking for a best functional relationship between x and y. One difference between these is, for example, that BIC, other parameter choices being equal, will have a better correlation coefficient between model and data, and AIC will have better extrapolation error measured as y-value error for a given extrapolated x-value.

Point 3 is a sometimes statement under some conditions

  • when the data are very noisy (large $σ$);

  • when the true absolute values of the left-out parameters (in our
    example $β_2$) are small;

  • when the predictors are highly correlated; and

  • when the sample size is small or the range of left-out variables is small.

In practice, a correct form of an equation does not mean that fitting with it will yield the correct parameter values because of noise, and the more noise the merrier. The same thing happens with R$^2$ versus adjusted R$^2$ and high collinearity. That is, sometimes when a parameter is added adjusted R$^2$ degrades while R$^2$ improves.

I would hasten to point out that these statements are optimistic. Typically, models are wrong, and often a better model will enforce a norm that cannot be used with AIC or BIC, or the wrong residual structure is assumed for their application, and alternative measures are needed. In my work, this is always the case.

$\endgroup$
  • 1
    $\begingroup$ I am not sure you are answering the questions. I am aware of the general limitations of information criteria, but that is not what I am asking about. Moreover, I do not understand your point if AIC and BIC have the SAME number of parameters then the claim is that AIC will be better for prediction than BIC. When alternative models have the same number of parameters, AIC and BIC comparison boils down to comparing likelihoods, and both AIC and BIC will select the same alternative. Could you also elaborate what you mean by a better model will enforce a norm that cannot be used with AIC or BIC? $\endgroup$ – Richard Hardy Dec 10 '17 at 15:10
  • $\begingroup$ Continued: As long as we have the likelihood and the degrees of freedom, we can calculate AIC and BIC. $\endgroup$ – Richard Hardy Dec 10 '17 at 15:12
  • $\begingroup$ @RichardHardy True: As long as we have the likelihood and the degrees of freedom, we can calculate AIC and BIC. However, the calculation will be sub-optimal and misleading if the residuals are Student's-T and we have not used AIC and BIC for Student's-T. Unlike Student's-T, there are distributions of residuals for which ML may be unpublished, for example Gamma, Beta etc. $\endgroup$ – Carl Dec 10 '17 at 16:42
  • $\begingroup$ Thank you for the clarification! I believe there should exist an answer to the questions above that is quite simple and general. More specifically, I do not think it needs to involve "ugly" cases and failures of AIC and BIC. To the contrary, I feel there should be a rather basic case that could illustrate why the paradox is only apparent rather than real. At the same time, your second paragraph seems to go in the opposite direction. Not that it would not be valuable in itself, but I am afraid it could distract us from the real underlying questions here. $\endgroup$ – Richard Hardy Dec 10 '17 at 17:31
  • $\begingroup$ @RichardHardy Often the practical question is intractable to AIC. For example, comparison of the same or different models with differing norms and/or data transformations or analysis of complicated norms, e.g., error reducing Tikhonov regularization of a derived parameter, general inverses etc. This needs to be mentioned as well lest someone use AIC, BIC incorrectly. $\endgroup$ – Carl Dec 10 '17 at 19:05
1
$\begingroup$

I read Shmueli's "To Explain or to Predict" (2010) a couple of years ago for the first time and it was one of the most important readings for me. Several great doubts come to solve after such reading.

It seems me that the contradictions you notice are less relevant that it seems to be. I try to reply to your two questions together.

My main argument is that your point 3 do not appear at pag 307 (here there are the detail) but at the begin of the discussion – bias-variance tradeoff argument (par 1.5; in particular end of pag 293). Your point 3 is the core message of the article. (See EDIT)

Your points 1 and 2 are related to the sub-argument of model selection. At this stage the main important practical difference between explanatory and predictive models do not appear. The analysis of the predictive models must involve out of sample data, in explanatory models it is not the case.

In predictive framework, firstly we have model estimation, then model selection that is something like evaluate the model (hyper)parameters tuning; at the end we have model evaluation on new data.

In explanatory framework, model estimation/selection/evaluation are much less distinguishable. In this framework theorethical consideration seems me much more important that the detailed distinction between BIC and AIC.

In Shamueli (2010) the concept of true model is intended as theoretical summary that imply substantial causal meaning. Causal inference is the goal. [For example you can read: “proper explanatory model selection is performed in a constrained manner … A researcher might choose to retain a causal covariate which has a strong theoretical justification even if is statistically insignificant.” Pag 300]

Now, the role of true model in causal inference debate is of my great interest and represent the core of several question that I opened on this web-community. For example you can read:

Regression and causality in econometrics

Structural equation and causal model in economics

Causality: Structural Causal Model and DAG

Today my guess is that the usual concept of true model is too simplistic for carried out exhaustive causal inference. At the best we can interpret it as very particular type of Pearl’s Structural Causal Model.

I know that, under some condition, BIC method permit us to select the true model. However the story that is behind this result sound me as too poor for exhaustive causal inference.

Finally the distinction between AIC and BIC seems me not so important and, most important, it does not affect the main point of the article (your 3).

EDIT: To be clearer. The main message of the article is that explanation and prediction are different things. Prediction and explanation (causation) are different goal that involve different tools. Conflation between them without understood the difference is a big problem. Bias-variance tradeoff is the main theoretical point that justify the necessity of the distinction between prediction and explanation. In this sense your point 3 is the core of the article.

EDIT2 In my opinion the fact here is that the problems addressed by this article are too wide and complex. Then, more than as usual, concepts like contradiction and/or paradox should be contextualized. For some readers that reads your question but not the article can seems that the article at all, or at least in most part, should be refuse, until somebody do not resolve the contradiction. My point is that this is not the case.

Suffice to say that the author could simply skip model selection details and the core message could remain the same, definitely. In fact the core of the article is not about the best strategy to achieve good prediction (or explanation) model, but to show that prediction and explanation are different goal that imply different method. In this sense your point 1 and 2 are minor and this fact resolve the contradiction (in the sense above).

At the other side remain the fact that AIC bring us to prefer long rather then short regression and this fact contradicts the argument at your point 3 is refer to. In this sense the paradox and or contradiction remain.

Maybe the paradox come from the fact that the argument behind point 3, bias-variance trade-off, is valid in finite sample data; in small sample can be substantial. In case of infinitely large sample, estimation error of parameter disappear, but possible bias term no, then the true model (in empirical sense) become the best also in the sense of expected prediction error. Now the good prediction properties of AIC is achieved only asymptotically, in small sample it can select models that have too many parameters then overfitting can appear. In case like this is hard to say precisely in what way the sample size matters.

However in order to face the problem of small sample a modified version of AIC was developed. See here: https://en.wikipedia.org/wiki/Akaike_information_criterion#Modification_for_small_sample_size

I done some calculus as examples and if these are free of mistake:

for the case of 2 parameters (as the case in Shamueli example) if we have less than 8 obs AIC penalizes more than BIC (as you says). If we have more than 8 but less than 14 obs AICc penalizes more than BIC. If we have 14 or more obs BIC is again the more penalizer

for the case of 5 parameters, if we have less than 8 obs AIC penalizes more than BIC (as you says). If we have more than 8 but less than 19 obs AICc penalizes more than BIC. If we have 19 or more obs BIC is again the more penalizer

for the case of 10 parameters, if we have less than 8 obs AIC penalizes more than BIC (as you says). If we have more than 8 but less than 28 obs AICc penalizes more than BIC. If we have 28 or more obs BIC is again the more penalizer.

Finally let me remark that if we remain very close to author words we can read that she do not explicitly suggest to use AIC in prediction and BIC in explanation (as reported at your point 1). She essentially said that: in explanatory model theoretical consideration are relevant and in prediction no. This is the core of the difference between these two kind of model selection. Then AIC is just presented as “popular metric” and its popularity come from the idea behind it. We can read: “A popular predictive metric is the in-sample Akaike Information Criterion (AIC). Akaike derived the AIC from a predictive viewpoint, where the model is not intended to accurately infer the “true distribution,” but rather to predict future data as accurately as possible”.

$\endgroup$
  • $\begingroup$ Thank you for your answer! I read it a couple of times, but your answer is not obvious to me. You give a nice discussion, but I would like to see a clearer conclusion w.r.t. to my questions. I am also not sure Shmueli puts as much emphasis on causal models as you are indicating. In any case, I am still puzzled.... $\endgroup$ – Richard Hardy Feb 9 at 15:52
  • $\begingroup$ @RichardHardy; you are not the first that notice me doubt about causal meaning in Shamueli (2010) article. I taken seriously this doubt. However I do not have doubt more. In Shamueli (2010) explanation and causation are synonym, there are no place for doubt about this. $\endgroup$ – markowitz Feb 9 at 16:30
  • $\begingroup$ Read here: “We mention this type of modeling [descriptive] to avoid confusion with causal-explanatory and predictive modeling, and also to highlight the different approaches of statisticians and non-statisticians.” (pag 291). Moreover the words causal/causation/causality/cause, appears almost 100 times in the article!!! $\endgroup$ – markowitz Feb 9 at 16:30
  • $\begingroup$ I repeat my message here. Your point 3 are the core, 1 and 2 are secondary detail. For me this is enough for solve the contradiction in favor of point 3. In prediction parsimony is good. In linear regression example short regression is better. $\endgroup$ – markowitz Feb 9 at 16:31
  • $\begingroup$ I added an EDIT for clarification. $\endgroup$ – markowitz Feb 9 at 16:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.