2
$\begingroup$

I need a general guide on what are the appropriate approaches to automated feature selection in multiple regression with categorical variables.

In my case, I have several numeric and categorical independent variables. I want to predict a numeric value and I am going to make use of multiple regression, including these categorical variables according to the effect coding strategy (find effect coding ref. here).

My questions are:

  • I am familiar with stepwise feature selection methods that I used in logistic regression models. Are they likely to be successful in this case, too?

  • When is there a moment to apply such automated feature selection methods? I mean: if I run them after introducing effect-transformed variables, there is a possibility the method reject e.g. a part of effect-transformed variables, drawn from one categorical variable (this categorical variable is not fully represented then), isn't it? Is this a problem?

  • What are the most popular automated feature selection methods when dealing with categorical variables?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
5
$\begingroup$

Stepwise regression does not work well with logistic regression and I expect it to be equally unsuccessful here. What made you think you need feature selection as part of the modeling process?

If you absolutely do need to incorporate feature selection, choose a method that keeps together multiple parameters describing one predictor, such as $F$ tests with multiple numerator degrees of freedom or other simple translations of partial sums of squares.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Frank, thank you for your attention to this question. What made you think you need feature selection as part of the modeling process? <- this is because of my general belief that a large number of variables in the model (appearing especially, when I incorporate effect-transformed variables) leads to too complex models with bad model features (overfitting etc.) $\endgroup$
    – Marta Karas
    Commented Sep 29, 2014 at 12:17
  • 3
    $\begingroup$ Feature selection has nothing to do with solving that problem. It is a mirage. You are actually "spending degrees of freedom" to do variable selection, and the large amount of resulting model uncertain undoes apparent advantages of variable selection. Think about it this way: I know of only 2 unbiased estimates of $\sigma^2$ in ordinary linear models. One involves the number of parameters in a pre-specified model and the other involves estimating the effective number of parameters upon data mining. This effective number is closer to the number of original candidates than # significant. $\endgroup$
    – Frank Harrell
    Commented Sep 29, 2014 at 12:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.