3
$\begingroup$

I am trying to build a logistic regression model. I have some categorical variables for which I have created dummy variables (eg. Department). I also have some numeric variables like Age and Tenure.

My question is which of the approach should I use-

  1. Should I use a combination of dummy variables and numeric variables as an input to my logistic model.
  2. Or, should I create categories of numeric variables based on response rate and use these categories to create dummy variables for numeric variables as well.

In first approach I am afraid that I numeric variable will become highly significant and cause overfitting. Also, they will reduce the real "significance" of dummy variables. In second approach I am afraid that I will loose a lot of information.

$\endgroup$
2
$\begingroup$

Adding a numberic variable to a logistic regression is unlikely to lead to overfitting as it imposes quite a strong contraint: every unit increase in your numeric variable leads to an increase or decrease in the odds of success of a factor $\exp(\beta)$. By default this factor is constant, which is how you can describe that effect with just one number. You can relax that assumption by adding polynomials, splines, or breaking your numeric variable up into different categories. Overfitting starts to become an issue if you use a polynomial of too high order, too many knots or break your variable up in too many classes.

So if anything your strategy 2 is in danger of loosing too much information if you choose too few categories or overfitting if you choose too many categories.

$\endgroup$
  • $\begingroup$ Thanks Marteen, inclusion of numeric variable does not seem to overfit as you said. $\endgroup$ – Gaurav Singhal Jul 16 '15 at 12:28
2
$\begingroup$

I suggest that you take the first approach. With the second approach, not only you might lose some information, but you can also introduce biases in the dataset by categorizing numerical features.

To make the first approach work, you need to pre-process all the features once you create dummy variables, which includes:

  • Centering and scaling
  • Transformation to remove skewness in data
  • Remove highly correlated features
  • Remove features that have near-zero variance

To prevent overfitting, you can use penalized logistic regression model.

$\endgroup$
  • $\begingroup$ Thanks, I do follow the points 3rd and 4th. I think centering and scaling will do the trick. I will have to read up on penalized regression model ! $\endgroup$ – Gaurav Singhal Jul 15 '15 at 20:19
  • $\begingroup$ I accepted Marteen Answers as what he said is correct - Adding a numberic variable to a logistic regression is unlikely to lead to overfitting. $\endgroup$ – Gaurav Singhal Jul 16 '15 at 12:27
  • 1
    $\begingroup$ I do not think your last bullet point is a good idea. In any case, it conflicts with your first point: once standardized, all predictors have unit variance. There is also no reason to transform the predictors to remove skewness, predictor transormafions should be used to deal with issuses of model fit, which is fundamentally abou the relationship between x and y. $\endgroup$ – Matthew Drury Apr 7 '17 at 14:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.