I use logistic regression to model the probability of an event and all of my features are categorical variables. Note that some values of the categorical variables are more frequent than others. The categorical features are converted into boolean/dummy variables and then the data are pushed to logistic regression for training. Logistic regression provides a set of coefficients for every of dummy variable.
One of the problems that I see is that not all coefficients can be trusted equally. Intuitively I believe that I should "trust" coefficients of dummy variables with many occurrences in the dataset. The coefficients of rarely activated dummy variables are estimated by using only few data points and thus they can have a huge error.
My question is how can measure the trust of each coefficient? Testing the significance by using the value and the standard errors of the coefficients does not necessarily seems to incorporate the information on how rare a particular feature is?
Finally I am aware that I can perform feature selection before logistic regression (and I do), nevertheless I am interested in how one can model/detect the above strictly within the context of logistic regression.