13
$\begingroup$

I'm doing a simple AIC-based backward elimination model where some variables are categorical variables with multiple levels. These variables are modeled as a set of dummy variables. When doing backward elimination, should I be removing all the levels of a variable together? Or should I treat each dummy variable separately? And why?

As a related question, step in R handles each dummy variable separately when doing backward elimination. If I wanted to remove an entire categorical variable at once, can I do that using step? Or are there alternatives to step which can handle this?

Improve this question
$\endgroup$
4
  • $\begingroup$ I think it is hard to answer this question in a general context. If possible, can you pls provide some insights about what is your data? what do you want to infer? possible assumptions, etc. Providing these details might help us in answering your questions better. $\endgroup$
    – suncoolsu
    Commented Jan 7, 2011 at 1:02
  • $\begingroup$ @suncoolsu I'm not really at liberty to discuss any of that. If it's possible to answer the question with a decision tree ('if you're assuming X, then do Y'), that could be really helpful. $\endgroup$
    – nerdbound
    Commented Jan 7, 2011 at 2:11
  • 1
    $\begingroup$ For clarity, are you interested in interpreting the model as causal or are you simply interested in prediction? $\endgroup$
    – Andy W
    Commented Jan 7, 2011 at 2:18
  • $\begingroup$ Yes. I agree to Matt Parker. Moreover, if the cat data column has 5 entries, say, 1,2,3,4,5, the GLM model provides 4 columns for it, say, catCol1,catCol2,catCol3,catCol4, and if we decide to elliminate the catCol with the highest p-value, say catCol4, the rest three doesnt make much sense, as the data can always possess the value of catCol4. At such a situation, the GLM would be in confusion, and would show an error.. $\endgroup$
    – Ebby
    Commented Apr 13, 2018 at 4:01

2 Answers 2

Reset to default
8
$\begingroup$

I think you'd have to remove the entire categorical variable. Imagine a logistic regression in which you're trying to predict if a person has a disease or not. Country of birth might have a major impact on that, so you include it in your model. If the specific USAmerican origin didn't have any impact on AIC and you dropped it, how would you calculate $\hat{y}$ for an American? R uses reference contrasts for factors by default, so I think they'd just be calculated at the reference level (say, Botswana), if at all. That's probably not going to end well...

A better option would be to sort out sensible encodings of country of birth beforehand - collapsing into region, continent, etc. and finding which of those is most suitable for your model.

Of course, there are many ways to misuse stepwise variable selection, so make sure that you're doing it properly. There's plenty about that on this site, though; searching for "stepwise" brings up some good results. This is particularly pertinent, with lots of good advice in the answers.

Improve this answer
$\endgroup$
5
  • 1
    $\begingroup$ Thanks for the response. I guess my concern is that, in your example, if one country of birth is a fantastic predictor of a disease, and all the other countries of origin are poor predictors, then the backward elimination model will tell me to remove country of birth, even though country of birth is hardly useless as a covariate. In other words, it might make sense to have a single variable for 'are you Lithuanian?' even though country of birth as a whole does not predict well. Are there sensible ways to handle that sort of problem? $\endgroup$
    – nerdbound
    Commented Jan 7, 2011 at 1:53
  • 2
    $\begingroup$ (+1) If one country is a great predictor, then the whole group of countries will be significant and it will be retained. @Matt Parker is correct: keep groups of dummy variables together and base the entering and leaving criteria on the joint significance level for the entire group, not on the significance levels for the individual dummies. $\endgroup$
    – whuber ♦
    Commented Jan 7, 2011 at 2:18
  • 2
    $\begingroup$ @whuber OK one more potentially dumb comment from me and then I'll be at peace when I understand why I'm wrong :-P It seems to me that if I'm looking at AIC and I exclude, say, 200 country dummies as a group, the AIC will improve because the new model will have 200 fewer variables, and it will get worse only to the extent that those dummies were good predictors. If only one of the variables was a good predictor, it seems to me that on net the AIC will make the model with 200 fewer variables look better, but then I'll be missing out on that one variable... $\endgroup$
    – nerdbound
    Commented Jan 7, 2011 at 3:03
  • 4
    $\begingroup$ @nerdbound That's not dumb at all. However, if only one dummy in 200 works well, you really have a different categorical variable: it's a new binary (or reclassified) variable. @Matt Parker addresses this issue in his second paragraph. This looks like a matter for exploratory analysis (which is what stepwise regression is anyway). It's perfectly fine to torture your data in this way, but after all the smoke clears, you need to test your model on independent data. $\endgroup$
    – whuber ♦
    Commented Jan 7, 2011 at 3:09
  • 1
    $\begingroup$ This might be relevant when there is very many levels: stats.stackexchange.com/questions/146907/… $\endgroup$
    – kjetil b halvorsen ♦
    Commented Jan 24, 2020 at 0:59
0
$\begingroup$

As for the example of countries, I think if the dummy variable for a specific country is selected, then it means this country is a predictor in comparison with all other countries combined (no need to create a new binary variable). The problem I have very often is dummy variables that reflects, for example, the severity of a disease (such as -, +, ++, +++). Sometimes the dummy variable for ++ is selected but the dummy variable for +++ is not. In this case, reclassification might be useful.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.