3
$\begingroup$

I'm running a logistic regression with a selection of categorical predictors. I have split my data into a training and testing set to evaluate my model. One of these predictors is "employment" which has over 50 levels. For some of the levels of this factor, the amount of observations are extremely limited, as low as 1. the range of observations in each variable is also very large (1 - 2011). I have two questions regarding this issue:

1) It's not possible to represent all levels of employment in the training and the testing set due to the fact that for some levels there is only one observation. I don't want to exclude these levels because those categories are necessary. How can I include then in the model? I suspect that I should possibly create some more observations which include those categories that just consist of noise. If that is the solution, how many of such observations would be required?

2) How does the model deal with those categories that have such few observations including those with 1, 2 or 4? Their associated standard errors are inflated and their p values are far from significant, however their estimates can be relatively large. Will they have weird effects on my model or are they dealt with automatically?

Below is a sample of the frequency of observations per each level of the category.

Improve this question
$\endgroup$
3
  • 1
    $\begingroup$ This is a use case for regularization with fused lasso, see stats.stackexchange.com/questions/227125/… and links therein. $\endgroup$
    – kjetil b halvorsen ♦
    Commented May 16, 2017 at 10:22
  • 4
    $\begingroup$ Have you considered recoding the employment variable into broader categories? Using categories of employment is common practice and can still be relevant depending on the phenomenon of interest. Otherwise it is hard to see how you can infer much from categories with single observations. $\endgroup$
    – Wes
    Commented May 16, 2017 at 10:41
  • $\begingroup$ Combining the two ideas, you could encode into broader conceptual categories of employment, then use these as fixed effects in a mixed effect model that includes the original fine grained categories as random effects. This gets you a nice interpretable model, and give a bit of regularization to deal with overfitting from the rare class issue. $\endgroup$
    – Matthew Drury
    Commented Oct 24, 2017 at 15:17

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.