3
$\begingroup$

I am running logistic regression using glm in R on data, that has some indicator variables to it. Two of those have multiple levels and have been rewritten as (#levels-1) new dummy predictors respectively. As for variable selection, I use step (backward variable selection based on AIC), which is where I struggle:

Even though all dummies of the initial indicator were kept or removed, I wonder what one would do, if a single dummy (i.e. one of a set of associated dummies) was eliminated during the variable selection? After all, this would mean that I lose the information encoded for the base-level (since it is "indicated" by 0s in all the other dummies).

Could someone tell me what to do if such a case occurred?

Thanks & Regards

$\endgroup$

1 Answer 1

3
$\begingroup$

You shouldn't be rewriting your categorical data into dummies yourself. Just feed your categorical data into glm as-is; R will internally turn it into dummies. Then stepAIC() and similar methods can work correctly and either keep or remove the entire predictor, not only single levels.

Alternatively, you can look at whether collapsing different levels of your categorical variable improves matters, either using AIC or with Likelihood Ratio or Wald tests. As an example, let's create some toy data:

set.seed(1)
foo <- data.frame(dv=sample(x=c(0,1),size=100,replace=TRUE),
  iv=sample(LETTERS[1:5],size=100,replace=TRUE))
model <- glm(dv~iv,family="binomial",data=foo)
AIC(model)
[1] 140.7574

Next, we collapse a couple of levels of foo$iv into single levels of a new predictor and fit a model with the new predictor:

foo$new.iv <- NA
foo$new.iv[foo$iv %in% c("A","B")] <- "a"
foo$new.iv[foo$iv %in% c("C","D")] <- "b"
foo$new.iv[foo$iv == "E"] <- "c"
AIC(update(model,.~new.iv))
[1] 137.8161

AIC is lower with collapsed factor levels. This is not surprising, given that there is no relationship between foo$iv and the dependent variable, so reducing the degrees of freedom should always reduce AIC.

Of course, you should not simply test all possible combinations of factor levels for collapsing; that would be data dredging and even using AIC would invalidate subsequent inferences if you do not account for this process. Instead, let theory on how the data were generated guide you. And report what you did so consumers of your analysis can get a feeling how much iterative model fitting was involved.

$\endgroup$
7
  • $\begingroup$ Thanks for your reply, can't vote up or accept yet, but will do as soon as I can. Could you please elaborate on the collapsing part? Say I had 4 levels, would I simply combine a subset of levels to an artificial new one whilst keeping the others (subsequently enumerating through all proper subsets and subset-combinations of the indicator)? $\endgroup$
    – sim
    Commented Jan 18, 2013 at 10:21
  • $\begingroup$ I edited my answer; clearer now? $\endgroup$ Commented Jan 18, 2013 at 11:02
  • $\begingroup$ thanks a lot for the additional comments & the illustrative example! it's much clearer to me now. $\endgroup$
    – sim
    Commented Jan 18, 2013 at 11:44
  • $\begingroup$ come to think of it, i would have one last question if you don't mind:i think i get your point about data dredging, since the enumerative approach (testing all subsets & combinations) is an exhaustive in-sample optimization of which the result is imposed on new instances (for prediction). am i correct so far? in your example, quite clear that the reduction of dfs will reduce the AIC - accompanied by the knowledge about, as you said, how the data was generated. however, since i seldom know about that in practice, could you give an example when you would collapse factors? thanks again! $\endgroup$
    – sim
    Commented Jan 18, 2013 at 12:21
  • 1
    $\begingroup$ It all depends on your data, of course. One example which I encountered in real life: retailers run promotions (like multi-page flyers in people's mailboxes) to increase sales, and a promotion predictor may have levels like "ad on the first page of a flyer", "ad on the second page", "ad on the last page" and so on. However, it is unclear whether the flyer page actually makes a difference, so collapsing all levels into one level or into two levels "ad on first page" and "ad somewhere else" may be better in terms of parsimony and predictive power. $\endgroup$ Commented Jan 18, 2013 at 12:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.