18
$\begingroup$

Having read Galit Shmueli's "To Explain or to Predict" (2010) I am puzzled by an apparent contradiction. There are three premises,

  1. AIC- versus BIC-based model choice (end of p. 300 - start of p. 301): simply put, AIC should be used for selecting a model intended for prediction while BIC should be used for selecting a model for explanation. Additionally (not in the above paper), we know that under some conditions BIC selects the true model among the set of candidate models; the true model is what we seek in explanatory modelling (end of p. 293).
  2. Simple arithmetics: AIC will select a larger model than BIC for samples of size 8 or larger (satisfying $\text{ln}(n)>2$ due to the different complexity penalties in AIC versus BIC).
  3. The "true" model (i.e. the model with the correct regressors and the correct functional form but imperfectly estimated coefficients) may not be the best model for prediction (p. 307): a regression model with a missing predictor may be a better forecasting model -- the introduction of bias due to the missing predictor may be outweighted by the reduction in variance due to estimation imprecision.

Points 1. and 2. suggest that larger models may be better for prediction than more parsimonious models. Meanwhile, point 3. gives an opposite example where a more parsimonious model is better for prediction than a larger model. I find this puzzling.

Questions:

  1. How can the apparent contradiction between points {1. and 2.} and 3. be explained/resolved?
  2. In light of point 3., could you give an intuitive explanation for why and how a larger model selected by AIC is actually better for prediction than a more parsimonious model selected by BIC?
$\endgroup$
  • 2
    $\begingroup$ I don't get the paradox/contradiction. AIC is efficient (asymptotically minimizes the expected prediction error) and BIC is consistent (asymptotically selects the true order). Point 3) says that bias may be outweighted by variance. There is obviously no guarantee that one is better than the other in a certain sample. So your "paradox" appears to be that for a given sample, AIC may not be best for prediction, which no surprise. For your Q2: if the bias increase induced by BIC's smaller model is larger than the variance increase in AIC's larger, AIC is better. $\endgroup$ – hejseb Jan 25 '16 at 6:42
  • 2
    $\begingroup$ I would suggest that you look at the first chapters in "Model Selection and Model Averaging" by Nils Hjort and Gerda Claeskens, maybe that will clear things up. $\endgroup$ – hejseb Jan 25 '16 at 6:43
1
$\begingroup$

They are not to be taken in the same context; points 1 and 2 have different contexts. For both AIC and BIC one first explores which combination of parameters in which number yield the best indices (Some authors have epileptic fits when I use the word index in this context. Ignore them, or look up index in the dictionary.) In point 2, AIC is the richer model, where richer means selecting models with more parameters, only sometimes, because frequently the optimum AIC model is the same number of parameters model as BIC the selection. That is, if AIC and BIC select models having the SAME number of parameters then the claim is that AIC will be better for prediction than BIC. However, the opposite could occur if BIC maxes out with a fewer parameters model selected (but no guarantees). Sober (2002) concluded that AIC measures predictive accuracy while BIC measures goodness of fit, where predictive accuracy can mean predicting y outside of the extreme value range of x. When outside, frequently a less optimal AIC having weakly predictive parameters dropped will better predict extrapolated values than an optimal AIC index from more parameters in its selected model. I note in passing that AIC and ML do not obviate the need for extrapolation error testing, which is a separate test for models. This can be done by withholding extreme values from the "training" set and computing the error between the extrapolated "post-training" model and the withheld data.

Now BIC is supposedly a lesser error predictor of y-values within the extreme values of range of x. Improved goodness of fit often comes at the price of bias of the regression (for extrapolation), wherein the error is reduced by introducing that bias. This will, for example, often flatten the slope to split the sign of the average left verses right $f(x)-y$ residuals (think of more negative residuals on one side and more positive residuals on the other) thereby reducing total error. So in this case we are asking for the best y value given an x value, and for AIC we are more closely asking for a best functional relationship between x and y. One difference between these is, for example, that BIC, other parameter choices being equal, will have a better correlation coefficient between model and data, and AIC will have better extrapolation error measured as y-value error for a given extrapolated x-value.

Point 3 is a sometimes statement under some conditions

  • when the data are very noisy (large $σ$);

  • when the true absolute values of the left-out parameters (in our
    example $β_2$) are small;

  • when the predictors are highly correlated; and

  • when the sample size is small or the range of left-out variables is small.

In practice, a correct form of an equation does not mean that fitting with it will yield the correct parameter values because of noise, and the more noise the merrier. The same thing happens with R$^2$ versus adjusted R$^2$ and high collinearity. That is, sometimes when a parameter is added adjusted R$^2$ degrades while R$^2$ improves.

I would hasten to point out that these statements are optimistic. Typically, models are wrong, and often a better model will enforce a norm that cannot be used with AIC or BIC, or the wrong residual structure is assumed for their application, and alternative measures are needed. In my work, this is always the case.

$\endgroup$
  • 1
    $\begingroup$ I am not sure you are answering the questions. I am aware of the general limitations of information criteria, but that is not what I am asking about. Moreover, I do not understand your point if AIC and BIC have the SAME number of parameters then the claim is that AIC will be better for prediction than BIC. When alternative models have the same number of parameters, AIC and BIC comparison boils down to comparing likelihoods, and both AIC and BIC will select the same alternative. Could you also elaborate what you mean by a better model will enforce a norm that cannot be used with AIC or BIC? $\endgroup$ – Richard Hardy Dec 10 '17 at 15:10
  • $\begingroup$ Continued: As long as we have the likelihood and the degrees of freedom, we can calculate AIC and BIC. $\endgroup$ – Richard Hardy Dec 10 '17 at 15:12
  • $\begingroup$ @RichardHardy True: As long as we have the likelihood and the degrees of freedom, we can calculate AIC and BIC. However, the calculation will be sub-optimal and misleading if the residuals are Student's-T and we have not used AIC and BIC for Student's-T. Unlike Student's-T, there are distributions of residuals for which ML may be unpublished, for example Gamma, Beta etc. $\endgroup$ – Carl Dec 10 '17 at 16:42
  • $\begingroup$ Thank you for the clarification! I believe there should exist an answer to the questions above that is quite simple and general. More specifically, I do not think it needs to involve "ugly" cases and failures of AIC and BIC. To the contrary, I feel there should be a rather basic case that could illustrate why the paradox is only apparent rather than real. At the same time, your second paragraph seems to go in the opposite direction. Not that it would not be valuable in itself, but I am afraid it could distract us from the real underlying questions here. $\endgroup$ – Richard Hardy Dec 10 '17 at 17:31
  • $\begingroup$ @RichardHardy Often the practical question is intractable to AIC. For example, comparison of the same or different models with differing norms and/or data transformations or analysis of complicated norms, e.g., error reducing Tikhonov regularization of a derived parameter, general inverses etc. This needs to be mentioned as well lest someone use AIC, BIC incorrectly. $\endgroup$ – Carl Dec 10 '17 at 19:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.