3
$\begingroup$

I have one response variable and a large number (>100) of explanatory variables.

METHOD 1: I have completed one approach where the explanatory variables have been reduced via a PCA (in accordance with existing literature looking at this common set of explanatory variables), before modelling the reduced set of independent variables by ranking the AIC scores of different models (the DREDGE function in the MuMIN package, R), after the necessary data exploration (e.g. colinearity checks).

However, I would also like to do a separate analysis that explores the significance of each explanatory variable individually. Following this, I would like to compare the most reliable models and influential variables against those included in the previous analysis (i.e. considering if the PCA approach stripped out some potentially influential independent variables).

METHOD 2: I thought of applying multiple GLMs and GAMs(with some models likely to be both linear and non-linear) accounting for the individual influence of each independent variable. All models would then be ranked by the AIC score.

Is method 2 a good way of achieving this? I am aware that with so many explanatory variables, type I error is a big concern, but the AIC score would be used informatively to rank which independent variables exert the most control over the response variable. I have seen debates about using AIC in nonnested setups, but some people seem to suggest it could be used as part of a model building procedure (e.g. see an answer here Non-nested model selection). The assumptions of the top X amount of models (X being consistent with METHOD 1) would then be validated, with all variables being tested for colinearity before then modelling the reduced set of variables as conducted in the METHOD 1.

Thank you in advance for any help and advice.

Improve this question
$\endgroup$
2
  • $\begingroup$ Be careful when you try to "explore the significance of each explanatory variable individually". Some of your >100 explanatory variables are likely to be inter-correlated, so their individual levels of "significance" in this data sample might be substantially different from a different sample. PCA helps mask that problem, as correlated variables show up in similar principal components. So you might want to consider regularization methods instead. $\endgroup$
    – EdM
    Commented Jul 7, 2015 at 14:54
  • $\begingroup$ Indeed I will be experiencing high levels of collinearity with some of the variables being derived from similar calculations. This was intended to be an alternative way of reaching a final set of candidate variables by exploring their individual significance against the response variable, with the PCA approach being shown to exclude some of the important ones. Following the ranking of each model by the AIC score, I was going to validate the assumptions and then test for colinearity before putting these into the final model as in Method 1. Is this not wise? Thank you for the comment! $\endgroup$
    – James White
    Commented Jul 7, 2015 at 15:18

3 Answers 3

Reset to default
2
$\begingroup$

Even if AIC or AIC(c) can be used for selecting among non-nested models, using them in the type of model building that you propose is not the best way to accomplish your goals reliably.

With the multicollinearity you have among your predictors, simple attempts to select particular predictors are highly unlikely to generalize well to the underlying population, regardless of how well they fit your particular sample. The particular predictors that work with your sample might not work so well on a subsequent sample. The ability to generalize can be tested, for example, by cross-validation or bootstrapping techniques. See An introduction to Statistical Learning for ways to proceed. My guess is that the approaches in your question, except for the original PCA, would not generalize well.

PCA can be a very good way to deal with multicollinearity, as correlated variables will tend to be in the same components. Regularization methods can also be used to help make models more generalizable. For example, the ridge regression method is very similar to PCA, where you essentially place a set of different weights on the principal components, instead of the all-or-none selection of components in standard PCA. If you need to select a subset of predictors, the LASSO has a better chance of doing so reliably. If your ultimate goal is prediction, note that you are probably better off erring on the side of keeping too many rather than too few predictors.

Should you use such reliable model building methods and want to compare different types of models then, perhaps AIC or AIC(c) would be useful. But don't use the model-building approach suggested in your original question.

Improve this answer
$\endgroup$
0
3
$\begingroup$

AIC roughly measures the distance between the true data generating process and a given model, so there's no reason why one couldn't compare AIC values between non-nested models.

According to Anderson and Burnham, this is a common myth about AIC. (http://warnercnr.colostate.edu/~anderson/PDF_files/AIC%20Myths%20and%20Misunderstandings.pdf)

Improve this answer
$\endgroup$
1
  • $\begingroup$ Thank you for the response! This has proved to be very helpful! $\endgroup$
    – James White
    Commented Jul 7, 2015 at 14:40
2
$\begingroup$

Per Burnham & Anderson (2002 §2.11.1), as long as the models are being fit to the exact same data (no removing outliers, no grouping or binning) then AIC(c) can be used to compare those models. One point they make in §2.11.4, specifically about regression, is that if the two models have different error structures, then they should be compared through a likelihood framework, and the shortcut of comparing RSS / $\hat{\sigma}^2$ should not be used.

Improve this answer
$\endgroup$
3
  • $\begingroup$ Thank you for the response. So, in this case my methodology is valid? Also, what benefits does AIC(c) offer over AIC? $\endgroup$
    – James White
    Commented Jul 7, 2015 at 14:42
  • $\begingroup$ Burnham & Anderson (2002 §2.4) recommend using AICc when the ratio of $n$ to $k$ (number of variables) is small ($\frac{n}{k} < 40$ or so), as the bias-correction estimate becomes more important. If $\frac{n}{k} $ is large, then AICc approaches AIC anyway. One caveat, when comparing models, one has to consistently use AICc or AIC, not mix them. Personally, I always use AICc. $\endgroup$
    – Avraham
    Commented Jul 7, 2015 at 15:07
  • $\begingroup$ I would have a low n to k ratio so I will look at that chapter in Burnham and Anderson (2002) and look to use that. Thank you. $\endgroup$
    – James White
    Commented Jul 7, 2015 at 15:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.