How many candidate models to include in AIC model selection?

Question

Is there a rule of thumb, perhaps related to sample size, for how many models to include in AIC model selection? Too many may seem like fishing while too few would be insufficient. I'm familiar with the guideline of not more than one predictor variable per 10 measures, is there something similar for how many candidate models one should create?

I've based my candidate models off hypotheses, but the number of candidate models quickly grows when I also model each variable separately, reduce my top performing models in various ways, and throw in some candidate versions including random effects.

In my particular case, I am comparing logistic regression models using different combinations of the same 13 covariates, some including a random effect of individual and some not. My sample size is 520. I'm looking for advice for comparing models of the same general form

Answer 1

For some general discussion of using AIC to select among a set of models, see this page. Frank Harrell's answer there says:

Generally speaking, AIC works best if used to select a unique single parameter (e.g., shrinkage coefficient) or to compare 2 or 3 candidate models.

In the specific case of logistic regression, however, comparing models based on different subsets of predictors as you propose can pose particular problems. Logistic regression has an inherent omitted-variable bias. If predictors that are related to class membership are not included in a model, the coefficients for other predictors will be biased. The answer shows in detail, for the related probit model, why this is so.

In logistic regression, unlike linear regression, such bias can occur even if the omitted predictors are uncorrelated to the included predictors, and the bias will be toward lower absolute values of coefficients than their true values. Omitting predictors thus might make it more difficult for you to identify truly significant relations of predictors to class membership.

In your case, with 13 predictors and 520 cases, you might be able simply to use a full model with all predictors and evaluate directly the significance of each predictor with respect to outcome. The rule of thumb is 10-20 cases of the least prevalent class per predictor evaluated, so if your smallest class has at least 25% prevalence, all predictors are continuous or binary, and you aren't including interactions, you should be OK. That approach should reduce or even eliminate further AIC-based comparisons.

If for some reason you need a more parsimonious model that omits some of your 13 predictors, you could consider stepwise backward elimination from the full model or a penalized approach such as LASSO. Finally, you should be evaluating your entire model-building process by cross validation, not just the performance of your final selected model. See this answer for more details.