2
$\begingroup$

Suppose I am modelling a binary outcome $y$ with $n$ covariates. If this model has the lowest AIC among all the other ones...would this be the one to use? Also in dealing with categorical variables, should you delete all levels or none? For example suppose a categorical variable $z$ has the following levels: $z_1, z_2$ and $z_3$. If $z_1$ is the reference level then I include $z_2$ and $z_3$ in the model. If I don't want to include the variable $z$ in the model....would I delete both $z_2$ and $z_3$? Or could I include one of them?

$\endgroup$
3
$\begingroup$

You are implying that it is good to remove variables/levels from a model. Using $Y$ to help decide how to model $X$ results in biased $\beta$s, standard errors that are too small, $P$-values that are too small, and confidence intervals that are too narrow. In the case of OLS it also results in an estimate of $\sigma^2$ that is too small. Removing levels of categorical predictors is another multiple comparisons problem. Think about doing an ANOVA on a 5-level factor then doing $t$-tests to decide which levels to pool, then doing an ANOVA on 3 levels (this is almost exactly what you are implying should be done through the use of dummy variables). The first ANOVA has a perfect multiplicity adjustment built into it, and the second will not come close to preserving type I error.

$\endgroup$
  • $\begingroup$ Suppose you are interested in covariate $x$ effect on $y$. Obviously you would include it in the model. But suppose that you also had four other variables in the model. In SAS would it be good to use selection = stepwise? $\endgroup$ – Thomas Nov 20 '11 at 17:22
  • $\begingroup$ I should let @frank-harrell answer, since you asked him, but you may also want to look at this: stats.stackexchange.com/questions/18638/… although it does not use the phrase, stepwise selection is what is being discussed there. $\endgroup$ – gung - Reinstate Monica Nov 20 '11 at 18:25
  • $\begingroup$ The fact that you are interested in examining the effect of $X$ on $Y$ doesn't have much to do with removing variables from a model, in my opinion. And this has nothing to do with which software you are using. $\endgroup$ – Frank Harrell Nov 20 '11 at 19:30
2
$\begingroup$

The AIC (and other, similar metrics) cannot tell you definitively which model to use, but is a very helpful guideline. In general, if the AIC for the full model is lowest, and if the AIC for other possible models is not really in the same ballpark, then there is a strong case to be made that the full model is the appropriate one.

As for factors, a factor with l levels burns l-1 degrees of freedom. These levels / parameters should be treated as a unit (e.g., tested together when testing a nested model without that factor vs. the model with that factor). If you decide to drop a factor from the model, you drop all the levels. Since a factor is represented by l-1 dummy coded vectors, that means you drop those l-1 predictors from the model, in your example, z2 and z3.

$\endgroup$
  • $\begingroup$ Why should all levels of a factor be dropped simultaneously? I can think of plenty of situations where only one level of a factor plays an important role in a model. $\endgroup$ – fmark Nov 20 '11 at 8:39
  • $\begingroup$ Generally it is good practice to keep all or none, and to resort to "none" with great care. It is better to pool levels using subject matter knowledge or by seeing which levels have the lowest prevalences. $\endgroup$ – Frank Harrell Nov 20 '11 at 14:29
  • $\begingroup$ @fmark I tried to keep this answer simple, to be most helpful for Thomas, but there is much more that can be said about that. I agree with Frank Harrell's comment. Now, cases where pooling levels is appropriate are certainly thinkable. 2 issues: (1) how do you know when your case is one of those? and (2) how do you pool? Dropping a dummy variable when using reference cell coding means pooling that level with the reference cell's level. However, it could easily be that e.g. pooling 2 non-reference cells is what's appropriate. These thorny issues are too big for a comment or even my answer. $\endgroup$ – gung - Reinstate Monica Nov 20 '11 at 17:54
  • $\begingroup$ @FrankHarrell I've broken this discussion out into a new question. $\endgroup$ – fmark Nov 21 '11 at 22:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.