1
$\begingroup$

I have many predictor variables (100+) and many outcome variables (10+). I am attempting to optimise which predictor variables to use when modelling my outcome variables. I am using R for the analysis and the outcome variables must be modeled separately using the glm function.

My issue is that the GLMs should all use the same predictor variables, and so when I am choosing which variables to use, I have to take all models into consideration and not just one. Since there are so many predictor variables, I am using backward selection for simplicity. If I were optimising a single model, I would minimise the AIC. However, since I have more than one model, I can't minimise the AIC for just one of them as the AIC for the others might be really high.

My question is this: When optimising more than one model, what value should I minimise at each iteration?

I believe I read somewhere that you are to take the sum of the AICs, but I can't find where I read that (sources would be incredibly helpful). I have a niggling feeling however that it might be beneficial to minimise the product of the AICs as that would reduce influence of any outlying values (this boils down to minimising the arithmetic vs geometric mean)

P.S. If anyone can suggest a different information criterion (such as BIC, etc.), how to use it in this case and why it is superior I would be more than happy to have the feedback.

$\endgroup$
  • $\begingroup$ For independent data sets adding would be the correct way of getting the overall AIC of the collection of models. $\endgroup$ – Glen_b -Reinstate Monica Aug 10 '16 at 10:17
  • $\begingroup$ @Glen_b In my opinion the output variables should be independent. I think this requirement is not satisfied by definition. $\endgroup$ – Cagdas Ozgenc Aug 10 '16 at 11:12
  • 1
    $\begingroup$ @Cagdas I mentioned that independence requirement already; in the absence of independence the sum would generally not be suitable. $\endgroup$ – Glen_b -Reinstate Monica Aug 10 '16 at 11:41
  • $\begingroup$ @Glen_b You said independent data sets. I am not sure what that even means. If the output variables are independent then the multivariate distribution will factor and then the log likelihood can be written as sum however summing AICs will mean 2k component will be summed several times as well which I think is not correct. We just need to add 2k only once on top of the sum of negative log likelihoods from the marginals. $\endgroup$ – Cagdas Ozgenc Aug 10 '16 at 12:31
  • $\begingroup$ Another way to do it, would be to perform some kind reduction technique on the predictors (e.g. factor analysis), or perhaps use an PLS regression. You'd get to control how much of the original variation you want to keep, and it'd solve your uniformity issue, as you can simply keep the same factors across your models. The modeling choice depends somewhat on your substantive setting, as with inferential analysis factors may pose problems with interpretation. Alternatively, you could use the mean change in adjusted R² across all models for each variable as a criterion. $\endgroup$ – Maxim.K Aug 10 '16 at 12:36
5
$\begingroup$

A stepwise procedure using AIC is a very bad idea as it is equivalent to variable selection/elimination based on a p-value.

A much better method is using expert knowledge to choose variables that you know should be included or excluded, paying careful attention to possible confounders and mediators, in conjunction with a shrinkage method such as LASSO.

$\endgroup$
  • $\begingroup$ That is true, but it can be quite a daunting task to make careful selections among hundreds of variables. $\endgroup$ – Maxim.K Aug 10 '16 at 12:41
  • 1
    $\begingroup$ @Maxim.K it may be daunting, but an effort should still be made. $\endgroup$ – Robert Long Aug 10 '16 at 12:43
  • $\begingroup$ If you work in an academic setting, and are making inferences, this might be doable in practice. If you work as a business analyst (and don't have weeks to develop just one aspect of your analysis), then this is a highly impractical suggestion. Also, in predictive modeling you wouldn't be concerned as much with this kind of selection process. $\endgroup$ – Maxim.K Aug 10 '16 at 12:57
  • 1
    $\begingroup$ @Maxim.K I don't like to make assumptions about how much time the OP has. $\endgroup$ – Robert Long Aug 10 '16 at 13:07
  • $\begingroup$ I never said anything about the OP. I stated that there are different analytical situations, and in some of those variable selection on substantive grounds is not practically feasible. In these situations one is forced to make model-driven selections. And the OP is about model-driven selection, as undesirable as it may be, at least I read it that way. $\endgroup$ – Maxim.K Aug 10 '16 at 14:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.