1
$\begingroup$

Given that the sample size of a VAR (or a similar model: VARX, SVAR etc.) reduces by $1$ for each extra dependent-variable lag that I introduce (since we need to drop the empty rows, or NaN's in programming speak), is the AIC an appropriate measure for model selection?

To be more specific, in this response to a previous question of mine, it was kindly pointed out that the dependent variable of the model needs to be the same across the models that are being compared for the AIC to be valid. Well, although it is true that the dependent variable stays the the same, the data does change because of the reduced sample size. Does this matter? (It is clear that this likely does not make a difference in the large-sample limit, but what about in a small sample?)

Finally, are any of the points above good reasons to consider using the AICc over the AIC for model selection?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

No, if your number of records changes, AIC is not appropriate. There's multiple idea for fixing this:

  1. deal with the missing informaiton to have the same number of records so that missing lags are no longer an issue (e.g. multiple imputation, single imputation of some form perhaps with a missing flag for those records, etc. or use an algorithm that intrinsically can deal with missingness like xgboost)
  2. use the smallest subset you can use all methods for (if you don't want to do (1) and still want to compare using AIC)
  3. do not use AIC (e.g. use cross-validation or some past-vs.-future split if this is a time series problem, i.e. not evaluating on the training data).

AIC vs. AICc is sort of irrelavant for this topic. AICc is meant to be better for small sample situations (although there seems to be disagreement on whether it truly is).

Finally, there's the question whether model selection is a sensible thing to do (esp. if the models are so close to each other that it's not really possible to be sure whether one is truly better than the other) or whether you want model averaging instead.

Improve this answer
$\endgroup$
6
  • $\begingroup$ Thank you for your answer! Would you be able to expand on/provide a reference for the statement "if your number of records changes, AIC is not appropriate"? I would like to build some intuition as to why that is. I also find it curious that model selection via AIC (or some IC) seems to be the standard in the VAR literature, with this topic swept under the rug... $\endgroup$
    – Anthony
    Commented Aug 23, 2021 at 9:12
  • 1
    $\begingroup$ E.g. section 2.11.1 of Burnham and Anderson's "Model selection and multimodel inference" book is titled "AIC Cannot Be Used to Compare Models of Different Data Sets" and reads:"Models can be compared using the various information criteria, as estimates of relative, expected K-L information, only when they have been fitted to exactly the same set of data. For example, if nonlinear regression model g1 is fitted to a data set with n = 140 observations, one cannot validly compare it with model g2 when 7 outliers have been deleted, leaving only n = 133." $\endgroup$
    – Björn
    Commented Aug 23, 2021 at 9:31
  • 1
    $\begingroup$ In short, the -2 log (likelihood) term in AIC needs to be based on the same set of observed and fitted values in all models you compare. One simple example to explain this is: Let's imagine the data point you use or don't use because of what lags are not available is the only data point that is really hard to fit. When you include it in AIC, the AIC looks bad, if you exclude it, then AIC looks good. However, this happens because it's a difficult data point, not because the model is somehow better with an extra lag added. There's also the problem of -2log(L) getting smaller if you omit records. $\endgroup$
    – Björn
    Commented Aug 23, 2021 at 9:36
  • 3
    $\begingroup$ @Anthony For VAR models specifically, I wouldn't say this is "swept under the rug"; the standard way to do it is to first pick a maximum lag $p$ and then compute AIC for VAR(1) to VAR($p$) while dropping the same first $p$ observations from each, so that exactly the same observations are used in all compared models (i.e. idea #2 in this answer). $\endgroup$
    – Chris Haug
    Commented Aug 23, 2021 at 11:20
  • 3
    $\begingroup$ @ChrisHaug Thanks for this, I'd missed this subtlety. It helps clear up a confusion I'd had, namely, that functions like select_order in statsmodels seemed to give different answers for optimal lag depending on the maximum number of lags allowed. $\endgroup$
    – Anthony
    Commented Aug 23, 2021 at 11:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.