Is it feasible to rank several unnested Generalized (Additive and Linear) Models by AIC score?

Question

I have one response variable and a large number (>100) of explanatory variables.

METHOD 1: I have completed one approach where the explanatory variables have been reduced via a PCA (in accordance with existing literature looking at this common set of explanatory variables), before modelling the reduced set of independent variables by ranking the AIC scores of different models (the DREDGE function in the MuMIN package, R), after the necessary data exploration (e.g. colinearity checks).

However, I would also like to do a separate analysis that explores the significance of each explanatory variable individually. Following this, I would like to compare the most reliable models and influential variables against those included in the previous analysis (i.e. considering if the PCA approach stripped out some potentially influential independent variables).

METHOD 2: I thought of applying multiple GLMs and GAMs(with some models likely to be both linear and non-linear) accounting for the individual influence of each independent variable. All models would then be ranked by the AIC score.

Is method 2 a good way of achieving this? I am aware that with so many explanatory variables, type I error is a big concern, but the AIC score would be used informatively to rank which independent variables exert the most control over the response variable. I have seen debates about using AIC in nonnested setups, but some people seem to suggest it could be used as part of a model building procedure (e.g. see an answer here Non-nested model selection). The assumptions of the top X amount of models (X being consistent with METHOD 1) would then be validated, with all variables being tested for colinearity before then modelling the reduced set of variables as conducted in the METHOD 1.

Thank you in advance for any help and advice.

Answer 1

Even if AIC or AIC(c) can be used for selecting among non-nested models, using them in the type of model building that you propose is not the best way to accomplish your goals reliably.

With the multicollinearity you have among your predictors, simple attempts to select particular predictors are highly unlikely to generalize well to the underlying population, regardless of how well they fit your particular sample. The particular predictors that work with your sample might not work so well on a subsequent sample. The ability to generalize can be tested, for example, by cross-validation or bootstrapping techniques. See An introduction to Statistical Learning for ways to proceed. My guess is that the approaches in your question, except for the original PCA, would not generalize well.

PCA can be a very good way to deal with multicollinearity, as correlated variables will tend to be in the same components. Regularization methods can also be used to help make models more generalizable. For example, the ridge regression method is very similar to PCA, where you essentially place a set of different weights on the principal components, instead of the all-or-none selection of components in standard PCA. If you need to select a subset of predictors, the LASSO has a better chance of doing so reliably. If your ultimate goal is prediction, note that you are probably better off erring on the side of keeping too many rather than too few predictors.

Should you use such reliable model building methods and want to compare different types of models then, perhaps AIC or AIC(c) would be useful. But don't use the model-building approach suggested in your original question.

Answer 2

AIC roughly measures the distance between the true data generating process and a given model, so there's no reason why one couldn't compare AIC values between non-nested models.

According to Anderson and Burnham, this is a common myth about AIC. (http://warnercnr.colostate.edu/~anderson/PDF_files/AIC%20Myths%20and%20Misunderstandings.pdf)

Answer 3

Per Burnham & Anderson (2002 §2.11.1), as long as the models are being fit to the exact same data (no removing outliers, no grouping or binning) then AIC(c) can be used to compare those models. One point they make in §2.11.4, specifically about regression, is that if the two models have different error structures, then they should be compared through a likelihood framework, and the shortcut of comparing RSS / $\hat{\sigma}^2$ should not be used.

• Burnham, K. P. & Anderson, D. R. Model Selection and Multimodel Inference: A Practical Information-Theoretic Approach Springer Science+Business Media, Inc., 2002