4
$\begingroup$

I am confused trying to match different model selection strategies with different modelling objectives. (Unfortunately, my confusion is reflected in the length of the post. Please be patient.)

Model selection strategies:
1. Pick the model with the best-behaved residuals (i.e. residuals that appear the closest to $i.i.d.$, that is, have the lowest autocorrelations, least evidence of heteroskedasticity, etc.)
2. Pick the model with the lowest BIC
3. Pick the model with the lowest AIC

Modelling objectives:
A. Descriptive: Find patterns in the data and model them so that no patterns remain in the model residuals (perhaps a poor definition?).
B. Explanatory: minimize model bias as defined in Hastie et al. "Elements of Statistical Learning" (2009) p. 223, equation (7.9). Loosely speaking, find the model that would best approximate the true data generating process if we were able to estimate the model perfectly.
C. Predictive: minimize the expected prediction error for a new observation as in the reference above. That is in contrast to B because we have to account for imperfect estimation due to the limited sample size at hand.

Regarding B and C, see also Shmueli "To Explain or To Predict" (2011). Once again, to contrast B and C, note that under B there is no penalty on model complexity since we do not care about estimation precision; model bias is the only thing that matters. Meanwhile, for C both model bias and model estimation variance matter.


Based on intuition (and to some extent on Rob J. Hyndman's blog post "To Explain or To Predict" -- but I don't blame him for my own mistakes), I would guess that strategy 1. matches with objective A, 2. with B and 3. with C. (Honestly, my definition of A was partly based on 1., which may lead to biased inference.) However, I am far from certain... E.g. the match between 2. and B does not make sense as BIC penalizes model complexity while in B we should ignore it. But then again BIC will asymptotically select the true model from the candidate model set, if the true model is in the set.

Question: How do model selection strategies 1., 2., 3. match/mistacth modelling objectives A, B, C?


Here is a concrete example. Suppose I have a time series $\{x_t\}$ (a sample from 1 to T) and two models, AR(1) and AR(2), indexed as model 1 and model 2. Suppose further that BIC(1) < BIC(2) suggesting model 1 is "better". Suppose also that the residuals of model 2 are well behaved while the residuals of model 2 have statistically significant partial autocorrelation (PACF) at lag 2.

It appears that residual diagnostics are in conflict with BIC with respect to model selection. But perhaps there is no actual conflict and the two actually match different modelling goals? For example, model selection based on residual diagnostics matches the goal of descriptive modelling while BIC-based selection matches the goal of explanatory modelling?


Somewhat related posts are this and this.

$\endgroup$
  • $\begingroup$ What's your goal? Your 1-3 strategies are a tiny subset of model selection tools used in practice. Also 2 and 3 do basically the same thing. $\endgroup$ – Aksakal Nov 17 '15 at 15:27
  • $\begingroup$ My goal is more general than a concrete application as indicated in my Question. My goal today may be prediction but tomorrow be description and then later be explanation. Strategies 2 and 3 artificially seem to similar but have quite different justifications, as far as I know. Also, they seem to match very different modelling objectives, namely, B versus C. The cited blog post as well as Shmueli's paper have a more extensive discussion on those. $\endgroup$ – Richard Hardy Nov 17 '15 at 15:41
  • 1
    $\begingroup$ @Zach, I see you prefer American English in the beginning of the post but then start accepting the other spelling later on (first single "l", then double "l"). I rolled over your edit to make sure my spelling is consistent throughout the post. $\endgroup$ – Richard Hardy Nov 17 '15 at 15:44
  • $\begingroup$ @RichardHardy My bad! I didn't even realize there was a difference between the American and British spelling. Thanks for educating me. $\endgroup$ – Zach Nov 17 '15 at 20:12
  • $\begingroup$ @Zach, I am no good at English myself, just happen to know this particular peculiarity. I am glad you liked learning it, too. $\endgroup$ – Richard Hardy Nov 17 '15 at 20:13
0
$\begingroup$

I´m going to attempt an answer because I think I can bring some clarity to the questions: the answer might not match, in that case please ignore.

First AIC and BIC. These measures are in a way very similar. When fitting a model, adding parameters can reduce the error. Adding parameters can also lead to overfitting: this is the famous bias and variance trade-off. If not familiar, look it up. AIC and BIC are both measures that correct the error measure of a model for the number of parameters, preventing you from adding to many parameters and overfitting your data. One of the two is more conservative (I forgot which one.)

For the residuals. Some models give optimal parameter estimations if assumptions are met; for example normal i.i.d. and homoskedastic residuals. Not meeting the assumptions does not mean that the model is of no value, merely that the estimations might be off or that error estimates are not similar across observations. Looking at the residuals it might be possible to add a parameter that for example removes heteroskedasticity in the model. This would improve the model.

Putting it together very loosely: AIC and BIC are there to protect you from variance. Residual assumptions can give directions on how to assess or improve a model in a more qualitative way (some times removing bias).

These measures have to be balanced and are not necessarily mutually prevailing.

$\endgroup$
  • $\begingroup$ Thanks for the answer! I basically agree with your arguments, which could be used to set the stage for my question. Unfortunately, it does not really answer it (or at least the question that I actually have, in case I was not able to formulate it well). I wish some more concreteness and clarity, and to get the actual question answered. By the way, BIC is more conservative than AIC for sample size of 8 and larger; you may edit the post to fix this. $\endgroup$ – Richard Hardy Nov 23 '15 at 9:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.