1
$\begingroup$

Is there a rule of thumb, perhaps related to sample size, for how many models to include in AIC model selection? Too many may seem like fishing while too few would be insufficient. I'm familiar with the guideline of not more than one predictor variable per 10 measures, is there something similar for how many candidate models one should create?

I've based my candidate models off hypotheses, but the number of candidate models quickly grows when I also model each variable separately, reduce my top performing models in various ways, and throw in some candidate versions including random effects.

In my particular case, I am comparing logistic regression models using different combinations of the same 13 covariates, some including a random effect of individual and some not. My sample size is 520. I'm looking for advice for comparing models of the same general form

$\endgroup$
  • $\begingroup$ Could you please edit your question to say a bit more about the relationships among your different models: in particular, are they of the same general form but including different predictors, or are they completely different types of models? Note that there may be some dispute about whether AIC comparisons work with non-nested models; see the answers on this page. $\endgroup$ – EdM Nov 28 '18 at 20:05
  • $\begingroup$ Since your dataset is so small, you can easily assess the predictive ability of your models directly using leave-one-out cross-validation rather then AIC. In that case, there is no real limit for the number of models you can compare. $\endgroup$ – sega_sai Nov 28 '18 at 21:01
  • $\begingroup$ I'm going to assess my top-performing models with k-fold cross validation as a second step, but as is customary in my field I need to first select models using AIC. I'm just looking for an appropriate number of candidates to present in my AIC table $\endgroup$ – Emily Nov 28 '18 at 21:07
0
$\begingroup$

For some general discussion of using AIC to select among a set of models, see this page. Frank Harrell's answer there says:

Generally speaking, AIC works best if used to select a unique single parameter (e.g., shrinkage coefficient) or to compare 2 or 3 candidate models.

In the specific case of logistic regression, however, comparing models based on different subsets of predictors as you propose can pose particular problems. Logistic regression has an inherent omitted-variable bias. If predictors that are related to class membership are not included in a model, the coefficients for other predictors will be biased. The answer shows in detail, for the related probit model, why this is so.

In logistic regression, unlike linear regression, such bias can occur even if the omitted predictors are uncorrelated to the included predictors, and the bias will be toward lower absolute values of coefficients than their true values. Omitting predictors thus might make it more difficult for you to identify truly significant relations of predictors to class membership.

In your case, with 13 predictors and 520 cases, you might be able simply to use a full model with all predictors and evaluate directly the significance of each predictor with respect to outcome. The rule of thumb is 10-20 cases of the least prevalent class per predictor evaluated, so if your smallest class has at least 25% prevalence, all predictors are continuous or binary, and you aren't including interactions, you should be OK. That approach should reduce or even eliminate further AIC-based comparisons.

If for some reason you need a more parsimonious model that omits some of your 13 predictors, you could consider stepwise backward elimination from the full model or a penalized approach such as LASSO. Finally, you should be evaluating your entire model-building process by cross validation, not just the performance of your final selected model. See this answer for more details.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.