8
$\begingroup$

In answer to a previous question factor pooling in model selection was discussed.

If a factor or categorical variable is to be dropped in model selection, should all levels be dropped simultaneously? If so, why?

The motivation for dropping factors is to aid model interpretation. For example, I might be interested in explaining the factors that influence customer behaviour when visiting a store and have a categorical variable "travel mode" with factors "walking, bus, private car, taxi, etc." In this context, I can remove all the dummy variables except "private car" because they have an insubstantial estimated magnitude and are not significant predictors of behaviour. I then end up with a "travelled in private car" vs "didn't travel in private car" variable and don't have to worry about troubling the reader with interpreting the other largely uninteresting variables.

Improve this question
$\endgroup$
7
  • 2
    $\begingroup$ What scientific sense would it make to retain some, but not all, levels of a factor? How would you interpret that? $\endgroup$
    – whuber ♦
    Commented Nov 22, 2011 at 3:48
  • 2
    $\begingroup$ +1 @fmark I'm glad you started a new question. I think this is the right format for your questions to be discussed to your satisfaction. Comments just aren't enough. $\endgroup$
    – gung - Reinstate Monica
    Commented Nov 22, 2011 at 6:49
  • $\begingroup$ @whuber I think what fmark may be thinking of is something like multiple comparison tests after a significant ANOVA; when you run orthogonal contrasts, you are essentially collapsing levels together. $\endgroup$
    – gung - Reinstate Monica
    Commented Nov 22, 2011 at 6:54
  • 1
    $\begingroup$ @rolando2 I don't think there was anything to strike me as nonsensical or uninterpretable: I was just asking. In some contexts dropping some levels of a factor makes no scientific sense and in others it can, as you indicate. Asking about interpretability is one way to approach fmark's question. $\endgroup$
    – whuber ♦
    Commented Nov 23, 2011 at 3:23
  • 1
    $\begingroup$ @whuber I've edited the question to explain why I might want to do this. $\endgroup$
    – fmark
    Commented Nov 23, 2011 at 3:36

1 Answer 1

Reset to default
1
$\begingroup$

I'm really not sure what the answer would be in the absence of crossvalidation. But if we are crossvalidating, and we find that, say, one ethnic group out of 6 is substantially different from the others wrt Y, I can't seem to see anything wrong with using only that group's dummy variable in the followup equation. If membership/nonmembership in that group, and none other, is helping to predict the outcome (or to explain it, for that matter), why gummy up the equation with a bunch of unhelpful predictor dummies, which would only figure to add noise to the prediction?

Improve this answer
$\endgroup$
4
  • 4
    $\begingroup$ Because removing insignificant terms only seems to help the predictions. It actually makes predictions worse in new data, and results in multiplicity problems that hurts type I error and confidence interval coverage. $\endgroup$
    – Frank Harrell
    Commented Nov 22, 2011 at 20:53
  • $\begingroup$ This conversation has been picked up again at stats.stackexchange.com/questions/24298/… $\endgroup$
    – rolando2
    Commented Mar 8, 2012 at 20:21
  • $\begingroup$ @FrankHarrell can you source this or show an example? People keep saying it's bad but I have never seen proof of it $\endgroup$
    – PJ_
    Commented Nov 18, 2022 at 14:28
  • $\begingroup$ hbiostat.org/rmsc Chapter 4 $\endgroup$
    – Frank Harrell
    Commented Nov 21, 2022 at 12:36

Not the answer you're looking for? Browse other questions tagged or ask your own question.