Model Selection: Logistic Regression

Suppose we have $n$ covariates $x_1, \dots, x_n$ and a binary outcome variable $y$. Some of these covariates are categorical with multiple levels. Others are continuous. How would you choose the "best" model? In other words, how do you choose which covariates to include in the model?

Would you model $y$ with each of the covariates individually using simple logistic regression and choose the ones with a significant association?

• In addition to my answer below (or others, if they emerge), the following has some good discussion of model selection (albeit not focused on logistic regression per se) stats.stackexchange.com/questions/18214/… – gung Nov 19 '11 at 20:36
• I'll quote @jthetzel from a recent comment on this site: "A good question, but one that most here studied in semester-long university courses, and some have spent careers studying." It's kind of like sitting down with a person and saying, "Can you teach me Swahili this afternoon?" Not that Gung doesn't make good points in his answer. It's just a vast territory. – rolando2 Nov 19 '11 at 20:44
• This is also a thread that, while for a very specific question, contains some advice from me generally: stats.stackexchange.com/questions/17068/… I'll also give my thoughts below. – Fomite Nov 19 '11 at 23:30
• Okay so I think I will just use AIC as a criterion. The full model has the lowest AIC. Also the AIC's are pretty different from each other. – Thomas Nov 21 '11 at 18:55

This is probably not a good thing to do. Looking at all the individual covariates first, and then building a model with those that are significant is logically equivalent to an automatic search procedure. While this approach is intuitive, inferences made from this procedure are not valid (e.g., the true p-values are different from those reported by software). The problem is magnified the larger the size of the initial set of covariates is. If you do this anyway (and, unfortunately, many people do), you cannot take the resulting model seriously. Instead, you must run an entirely new study, gathering an independent sample and fitting the previous model, to test it. However, this requires a lot of resources, and moreover, since the process is flawed and the previous model is likely a poor one, there is a strong chance it will not hold up--meaning that it is likely to waste a lot of resources.

A better way is to evaluate models of substantive interest to you. Then use an information criterion that penalizes model flexibility (such as the AIC) to adjudicate amongst those models. For logistic regression, the AIC is: $$AIC = -2\times\ln(\text{likelihood}) + 2k$$

where $k$ is the number of covariates included in that model. You want the model with the smallest value for the AIC, all things being equal. However, it is not always so simple; be wary when several models have similar values for the AIC, even though one may be lowest.

I include the complete formula for the AIC here, because different software outputs different information. You may have to calculate it from just the likelihood, or you may get the final AIC, or anything in between.

• I like AIC but beware that computing AIC on more than 2 pre-specified models results in a multiplicity problem. – Frank Harrell Nov 20 '11 at 14:31
• @FrankHarrell nice tip! – gung Nov 20 '11 at 17:29

There are many ways to choose what variables go in a regression model, some decent, some bad, and some terrible. One may simply browse the publications of Sander Greenland, many of which concern variable selection.

Generally speaking however, I have a few common "rules":

• Automated algorithms, like those that come in software packages, are probably a bad idea.
• Using model diagnostic techniques, like gung suggests, are a good means of evaluating your variable selection choices
• You should also be using a combination of subject-matter expertise, literature searchers, directed acyclic graphs, etc. to inform your variable selection choices.
• Well put, especially points 1 and 3. Model diagnostic techniques can result in a failure to preserve type I error. – Frank Harrell Nov 20 '11 at 14:32
• Well put @Epigrad. I would add one point though. Automated algorithms become very attractive when your problem becomes large. They may be the only feasible way of doing model selection in some cases. People are now analysing huge data sets with 1000s of potential variables and millions of observations. How is the subject matter's expertise at 1000-dimensional intuition? And what you will find is that even if you do it manually (i.e. with an analyst), they will likely end up creating some short-cut rules for choosing variables. The hard part is really coding up those choices. – probabilityislogic Nov 26 '11 at 11:22
• @probabilityislogic I would agree with that. Honestly, I think traditional techniques are poorly suited for very large data sets, but the tendency to fall back to more amenable techniques alarms me. If an automated algorithm can bias a data set with 10 variables, there's no reason it can't bias one with 10,000. The current emphasis on the acquisition of big data over its analysis in some parts makes me somewhat skittish. – Fomite Nov 27 '11 at 1:22
• @probabilityislogic In a deeply ironic twist, I now find myself working with a dataset with well over 10s of 1000s of potential variables >.< – Fomite Feb 15 '18 at 23:11

How would you choose the "best" model?

There isn't enough information provided to answer this question; if you want to get at causal effects on y you'll need to implement regressions that reflect what's known about the confounding. If you want to do prediction, AIC would be a reasonable approach.

These approaches are not the same; the context will determine which of the (many) ways of choosing variables will be more/less appropriate.