Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It only takes a minute to sign up.
Sign up to join this community
Suppose we have $n$ covariates $x_1, \dots, x_n$ and a binary outcome variable $y$. Some of these covariates are categorical with multiple levels. Others are continuous. How would you choose the "best" model? In other words, how do you choose which covariates to include in the model?
Would you model $y$ with each of the covariates individually using simple logistic regression and choose the ones with a significant association?
This is probably not a good thing to do. Looking at all the individual covariates first, and then building a model with those that are significant is logically equivalent to an automatic search procedure. While this approach is intuitive, inferences made from this procedure are not valid (e.g., the true p-values are different from those reported by software). The problem is magnified the larger the size of the initial set of covariates is. If you do this anyway (and, unfortunately, many people do), you cannot take the resulting model seriously. Instead, you must run an entirely new study, gathering an independent sample and fitting the previous model, to test it. However, this requires a lot of resources, and moreover, since the process is flawed and the previous model is likely a poor one, there is a strong chance it will not hold up--meaning that it is likely to waste a lot of resources.
A better way is to evaluate models of substantive interest to you. Then use an information criterion that penalizes model flexibility (such as the AIC) to adjudicate amongst those models. For logistic regression, the AIC is: $$ AIC = -2\times\ln(\text{likelihood}) + 2k $$
where $k$ is the number of covariates included in that model. You want the model with the smallest value for the AIC, all things being equal. However, it is not always so simple; be wary when several models have similar values for the AIC, even though one may be lowest.
I include the complete formula for the AIC here, because different software outputs different information. You may have to calculate it from just the likelihood, or you may get the final AIC, or anything in between.
There are many ways to choose what variables go in a regression model, some decent, some bad, and some terrible. One may simply browse the publications of Sander Greenland, many of which concern variable selection.
Generally speaking however, I have a few common "rules":
How would you choose the "best" model?
There isn't enough information provided to answer this question; if you want to get at causal effects on y you'll need to implement regressions that reflect what's known about the confounding. If you want to do prediction, AIC would be a reasonable approach.
These approaches are not the same; the context will determine which of the (many) ways of choosing variables will be more/less appropriate.
