# Trust of coefficients of Logistic Regression

I use logistic regression to model the probability of an event and all of my features are categorical variables. Note that some values of the categorical variables are more frequent than others. The categorical features are converted into boolean/dummy variables and then the data are pushed to logistic regression for training. Logistic regression provides a set of coefficients for every of dummy variable.

One of the problems that I see is that not all coefficients can be trusted equally. Intuitively I believe that I should "trust" coefficients of dummy variables with many occurrences in the dataset. The coefficients of rarely activated dummy variables are estimated by using only few data points and thus they can have a huge error.

My question is how can measure the trust of each coefficient? Testing the significance by using the value and the standard errors of the coefficients does not necessarily seems to incorporate the information on how rare a particular feature is?

Finally I am aware that I can perform feature selection before logistic regression (and I do), nevertheless I am interested in how one can model/detect the above strictly within the context of logistic regression.

• Why do you say "the standard errors of the coefficients does not necessarily seems to incorporate the information on how rare a particular feature is"? – Scortchi - Reinstate Monica Apr 16 '15 at 11:31
• This is my understanding from reading how the standard errors are estimated, I might be wrong. The estimation does not seem to include the frequency of the variable. Here is a reference: groups.google.com/d/msg/comp.soft-sys.stat.spss/Fv7Goxs_Bwk/… – amazedgaggle Apr 16 '15 at 11:44

First of all it is invalid to do feature selection in a way that is removed from modeling unless you are doing pure data reduction and not using $Y$.
You can probably improve on the approach by using penalized maximum likelihood estimation. See for example the R rms package lrm function. This will help with rare categories in estimating $\beta$.
• If the size of the dataset is too big to numerically handle stepwise regression, I don't understand how it is not too big to fit the full regression model. But to answer your question there is no real need to remove any variables. If you used $\alpha=0.5$ you would do little damage, and remove a few variables though. – Frank Harrell Apr 16 '15 at 14:59
• You don't really re-run the regression, you just take the output from the last step after removal of variables with $P > \frac{1}{2}$. Still now clear on how a full model fit uses less memory than stepwise. – Frank Harrell Apr 16 '15 at 15:34