2
$\begingroup$

I wonder whether it is sensible to use the following method of assessing relative importance of variables in a model.

  1. Compute a model and its $AIC$. Let's say it has $k$ predictors.
  2. Compute $k$ submodels, each of which has one predictor removed. Compute $AIC$ for all the submodels.
  3. Compute $k$ times $\Delta AIC$, which is defined as $AIC_{submodel} - AIC_{full~model}$.
  4. And then the relative importance of each variable is being quantified by the magnitude of its corresponding $\Delta AIC$.

EDIT. Just to clarify a few things: this approach is meant to be used in a specific situation, when one has a bunch of otherwise equally important predictors (that is, none of them is considered more important than others a priori) and want to figure out which of the predictors has the strongest relationship with the outcome. The idea is that the difference in AIC quantifies relative likelihood that one model minimizes information loss and not the other, so having $\Delta AIC$ defined for all predictors one can judge which of them minimize information loss the most.

So the aim here is to establish a relative hierarchy of predictors within a single model. Does this strategy make any sense?

Improve this question
$\endgroup$
  • $\begingroup$ Why not? You might review Ulrike Gromping's papers on the relative importance of variables in regression. She does a comprehensive review of the literature on this topic, e.g., prof.beuth-hochschule.de/groemping/relaimpo $\endgroup$ – Mike Hunter Jan 3 '17 at 19:06
  • $\begingroup$ I am concerned about the validity of this approach. $AIC$ is trying to find the model where the least information is lost from the "true" model (that generated the data). The $\Delta AIC$ is therefore something like a difference of KL divergences (obviously a huge oversimplification). I cannot see how the $\Delta AIC$ can then be compared w.r.t. the individual predictors. Not to mention that by removing a predictor, you are throwing away information from interactions, which may be necessary (I assume the predictors aren't all pairwise independent). $\endgroup$ – justanotherbrain Jan 3 '17 at 19:13
  • $\begingroup$ Actually I implicitly assumed here that there are no interactions, but predictors may be somewhat correlated. I think that this is a very typicla situation in many applications. $\endgroup$ – sztal Jan 4 '17 at 9:51
2
$\begingroup$

Burnham and Anderson have offered some guidelines on this. The best idea would probably be to look at their text book about information-based statistical inference:

Burnham, K. P.; Anderson, D. R. (2002), Model Selection and Multimodel Inference: A Practical Information-Theoretic Approach (2nd ed.), Springer-Verlag, ISBN 0-387-95364-7.

But I believe they talk about guidelines of using gaps of 2 as providing some evidence, and 4 providing stronger evidence, to prefer the larger model. There are more levels of evidence beyond that. They talk about this in various places, e.g. this paper:

http://www.ericlwalters.org/Burnham_etal_2011.pdf

Improve this answer
$\endgroup$
  • $\begingroup$ I'm not sure that this addresses the OP's question... The OP is trying to figure out which predictors are most important, but you are addressing the question of using more predictors versus fewer. $\endgroup$ – justanotherbrain Jan 3 '17 at 19:17
  • $\begingroup$ In the textbook and in that paper Burnham & Anderson also address the issue of general model selection using similar logic. Thanks. $\endgroup$ – not_bonferroni Jan 3 '17 at 20:08
  • $\begingroup$ As far as I understand the authors correctly, the approach I taken should be valid since deltaAIC quantifies the loss of information after throwing away a variables. So I accept this answer. Thanks! $\endgroup$ – sztal Jan 7 '17 at 18:21
  • $\begingroup$ Could you perhaps help with this follow-up question? stats.stackexchange.com/questions/349883/… $\endgroup$ – Tripartio Jun 5 '18 at 12:12
1
$\begingroup$

In addition to the great points @justanotherbrain mentioned, I'd like to add that "importance" of a variable is quite subjective and really depends on the application of the model. If it's an explanatory model where you are trying to make an inference on the association between an explanatory variable and the outcome, it's quite likely that some variables included in the model may not contribute very much in dropping the AIC. For example, if you want to test the association between a drug and a health outcome, but the drug is ineffective, the "drug" variable remaining in the model is unlikely to affect the AIC by much, but it is it the most important explanatory variable in the model.

In the case of predictive models, again, AIC may not be the appropriate metric to use, as each model that you build should be optimized specifically to the application of the model as well as the type of model being built. A model of continuous outcomes will have very different model fit statistics than a model of discrete outcomes.

There are actually existing well-established methods which may accomplish what you want. The LASSO developed by Rob Tibshirani is one where you could establish some sort of "variable importance hierarchy", and a more flexible approach then the one you describe, since each variable's parameter is regularized, rather than the binary status of being "in" or "out" of the model.

Improve this answer
$\endgroup$
  • $\begingroup$ I agree that in many cases the issue of "importance" of a variable may not boil down to simple increase of model accuracy after adding it (like in your example with a drug). However, my idea here was to get a simple measure of importance in a situation, when you have a equally theoretically (un)important predictors and you want to figure out which are in fact related to the outcome you're studying. I am gonna edit the question and add this remark. $\endgroup$ – sztal Jan 4 '17 at 9:54
  • $\begingroup$ Also, I am aware of the possibility of using LASSO or ELASTIC NET in this case. However, sometimes in data analysis there are some non-substantive layers like communication of results. Sometimes receivers are not accustomed with these newer techniques, so I wondered whether such an approach using more popular AIC measure may be valid. $\endgroup$ – sztal Jan 4 '17 at 10:02
  • $\begingroup$ In medical research other variable selection criteria aside from measuring prediction strength exist. One such approach is called qualitative interactions. This was developed by Lacey Gunter and her advisor Susan Murphy as her PhD dissertation. I published a paper with her on this method in the Pakistan Journal of Operations Research and Statistics and Lacey and Susan have an article in the Journal of Biopharmaceutical Statistics where I was guest editor. $\endgroup$ – Michael R. Chernick Jan 13 '17 at 3:41
  • $\begingroup$ Variable Selection for Qualitative Interactions in Personalized Medicine While Controlling the Family-Wise Error Rate. L. Gunter, J. Zhu and S,A, Murphy Journal of Biopharmaceutical Statistics (2011). $\endgroup$ – Michael R. Chernick Jan 13 '17 at 3:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.