1
$\begingroup$

I use logistic regression to model the probability of an event and all of my features are categorical variables. Note that some values of the categorical variables are more frequent than others. The categorical features are converted into boolean/dummy variables and then the data are pushed to logistic regression for training. Logistic regression provides a set of coefficients for every of dummy variable.

One of the problems that I see is that not all coefficients can be trusted equally. Intuitively I believe that I should "trust" coefficients of dummy variables with many occurrences in the dataset. The coefficients of rarely activated dummy variables are estimated by using only few data points and thus they can have a huge error.

My question is how can measure the trust of each coefficient? Testing the significance by using the value and the standard errors of the coefficients does not necessarily seems to incorporate the information on how rare a particular feature is?

Finally I am aware that I can perform feature selection before logistic regression (and I do), nevertheless I am interested in how one can model/detect the above strictly within the context of logistic regression.

Improve this question
$\endgroup$
2
  • $\begingroup$ Why do you say "the standard errors of the coefficients does not necessarily seems to incorporate the information on how rare a particular feature is"? $\endgroup$
    – Scortchi - Reinstate Monica ♦
    Apr 16, 2015 at 11:31
  • $\begingroup$ This is my understanding from reading how the standard errors are estimated, I might be wrong. The estimation does not seem to include the frequency of the variable. Here is a reference: groups.google.com/d/msg/comp.soft-sys.stat.spss/Fv7Goxs_Bwk/… $\endgroup$
    – amazedgaggle
    Apr 16, 2015 at 11:44

1 Answer 1

Reset to default
3
$\begingroup$

First of all it is invalid to do feature selection in a way that is removed from modeling unless you are doing pure data reduction and not using $Y$.

Standard errors automatically incorporate the distribution of predictors. Predictors with worse distributions (e.g., rare categories) give rise to larger standard errors.

You can probably improve on the approach by using penalized maximum likelihood estimation. See for example the R rms package lrm function. This will help with rare categories in estimating $\beta$.

Improve this answer
$\endgroup$
5
  • $\begingroup$ Thanks for your reply. Practically would you remove any variable with significance less than a threshold and rerun the regression to get the final coefficients? Stepwise regression is out of the question because of the sizes of the datasets. Thx! $\endgroup$
    – amazedgaggle
    Apr 16, 2015 at 12:37
  • $\begingroup$ If the size of the dataset is too big to numerically handle stepwise regression, I don't understand how it is not too big to fit the full regression model. But to answer your question there is no real need to remove any variables. If you used $\alpha=0.5$ you would do little damage, and remove a few variables though. $\endgroup$
    – Frank Harrell
    Apr 16, 2015 at 14:59
  • $\begingroup$ I was referring to the execution times rather than to memory footprint. So based on your reply, one can use a high alpha remove really noise variables and rerun the regression to estimate the final coefficients of the model. $\endgroup$
    – amazedgaggle
    Apr 16, 2015 at 15:27
  • $\begingroup$ You don't really re-run the regression, you just take the output from the last step after removal of variables with $P > \frac{1}{2}$. Still now clear on how a full model fit uses less memory than stepwise. $\endgroup$
    – Frank Harrell
    Apr 16, 2015 at 15:34
  • $\begingroup$ Got it thanks! Your reply and comments were very helpful. Concerning the memory thing, as I said I am NOT worried about the memory. I am worried about the execution time which will increase if I run multiple regressions with the Stepwise approach. Again thanks! :) $\endgroup$
    – amazedgaggle
    Apr 16, 2015 at 16:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.