4
$\begingroup$

I am trying to build a logistic regression model. I have some categorical variables for which I have created dummy variables (eg. Department). I also have some numeric variables like Age and Tenure.

My question is which of the approach should I use-

  1. Should I use a combination of dummy variables and numeric variables as an input to my logistic model.
  2. Or, should I create categories of numeric variables based on response rate and use these categories to create dummy variables for numeric variables as well.

In first approach I am afraid that I numeric variable will become highly significant and cause overfitting. Also, they will reduce the real "significance" of dummy variables. In second approach I am afraid that I will loose a lot of information.

Improve this question
$\endgroup$

2 Answers 2

Reset to default
3
$\begingroup$

Adding a numberic variable to a logistic regression is unlikely to lead to overfitting as it imposes quite a strong contraint: every unit increase in your numeric variable leads to an increase or decrease in the odds of success of a factor $\exp(\beta)$. By default this factor is constant, which is how you can describe that effect with just one number. You can relax that assumption by adding polynomials, splines, or breaking your numeric variable up into different categories. Overfitting starts to become an issue if you use a polynomial of too high order, too many knots or break your variable up in too many classes.

So if anything your strategy 2 is in danger of loosing too much information if you choose too few categories or overfitting if you choose too many categories.

Improve this answer
$\endgroup$
1
  • $\begingroup$ Thanks Marteen, inclusion of numeric variable does not seem to overfit as you said. $\endgroup$
    – Gaurav Singhal
    Commented Jul 16, 2015 at 12:28
2
$\begingroup$

I suggest that you take the first approach. With the second approach, not only you might lose some information, but you can also introduce biases in the dataset by categorizing numerical features.

To make the first approach work, you need to pre-process all the features once you create dummy variables, which includes:

  • Centering and scaling
  • Transformation to remove skewness in data
  • Remove highly correlated features
  • Remove features that have near-zero variance

To prevent overfitting, you can use penalized logistic regression model.

Improve this answer
$\endgroup$
3
  • $\begingroup$ Thanks, I do follow the points 3rd and 4th. I think centering and scaling will do the trick. I will have to read up on penalized regression model ! $\endgroup$
    – Gaurav Singhal
    Commented Jul 15, 2015 at 20:19
  • $\begingroup$ I accepted Marteen Answers as what he said is correct - Adding a numberic variable to a logistic regression is unlikely to lead to overfitting. $\endgroup$
    – Gaurav Singhal
    Commented Jul 16, 2015 at 12:27
  • 1
    $\begingroup$ I do not think your last bullet point is a good idea. In any case, it conflicts with your first point: once standardized, all predictors have unit variance. There is also no reason to transform the predictors to remove skewness, predictor transormafions should be used to deal with issuses of model fit, which is fundamentally abou the relationship between x and y. $\endgroup$
    – Matthew Drury
    Commented Apr 7, 2017 at 14:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.