1
$\begingroup$

My current understanding: Both AIC and BIC take the number of parameters as input when comparing nested models with a different number of parameters / predictors.

My question: Is it necessary / a good idea to adjust the number of parameters provided to AIC / BIC in case parameters are highly correlated (by design, as the predictors are gridpoint values of a (spatially correlated) atmospheric field)? Or should I put less trust into the AIC/BIC values in case the independence assumption of the underlying model is not fulfilled?

$\endgroup$
1
$\begingroup$

Your intuition is correct; you are describing the fundamental questions behind thinking about "effective degrees of freedom".

We should indeed put "less trust" in the AIC/BIC values when using correlated explanatory variables. Our AIC/BIC will potentially be negatively biased (i.e. we will assume that we have more explanatory variables that we actually have and penalise our information criterion more than expected). This happens because our model's complexity and the model's degrees of freedom may not correspond to each other closely. This "decoupling" has been first explored in Efron (1983) Estimating the error rate of a prediction rule: improvement on cross-validation. Janson, Fithian & Hastie (2015) give a short Effective degrees of freedom: a flawed metaphor

In general, when we have correlated predictors, the question of "correct" degrees of freedom is somewhat ill-defined. For example, in ridge regression the DoF are defined as: $\sum_i^p d_i^2 /(d_i^2 + \lambda)$ where $d_i = diag(D)$ from $X= UDV^T$ with $X$ being our design matrix and $UDV^T$ its singular decomposition while in the case of LASSO as the number of non-zero coefficients. Obviously, both of these procedures are affected by our choice of regularisation $\lambda$ which creates a circular situation where we are "choosing the hyper-parameters what give us the best IC but the IC itself then reflects how good are choice of hyper-parameters was", i.e. we go in circles. To that respect, as Karlsson et al. (2019) suggest in Performances of Model Selection Criteria When Variables are Ill Conditioned: "our final recommendation is that practitioners should not base their model building decisions only on the model selection criteria." I fully agree; IC use should inform but not fatalistically determine our model selection procedure.

(But do read the Karlsson et al. paper further, they do a nice simulation setting and show that the Hannan–Quinn information criterion is a better choice than $R^2$/AIC/BIC in "successfully identifying the true model" within their simulation settings.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.