3
$\begingroup$

I am working with 82 binary features constructed from six categorical features. I have about 1,600 observations. Some of these features correspond to extremely rare categories. Some of them have only one or two 1's in the entire sample; others have in the teens, and a few others have in the 20s and 30s.

There is a lot of natural variation in the data, so these very rare 1's are not likely to be informative. But how rare is "too rare?" The categories that appear only twice in 1,600 observations are probably too rare. But is 20 appearances still too rare? What principled approach can I take to choosing a rarity cutoff?

This also begs the question of what to do with the categories that are "too rare." I'm tempted to drop them. But given that they correspond to mutually exclusive blocks of indicators, that means I am effectively rolling the rare categories in with the baseline category. This isn't such a big deal for one or two observations, but I'm concerned about doing it with too many features at once.

Edit: I'm not just using these features in a predictive model. I'm also interested in the associations between features. I'm going to use either correlation or Cramer's V; I'd like to just drop a baseline category and compute the correlation matrix, but I imagine there are problems with that approach (and it's a whole separate question anyway).

$\endgroup$
3
$\begingroup$

Instead of using cut-off value for features, another possible way maybe using L1-norm regularization. If some rare features are not helpful, it's corresponding weight will be 0.

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ That's what I was already planning to do. I was concerned about causing numerical instability or somehow badly hurting efficiency. That won't be a problem in this case? $\endgroup$ – shadowtalker Apr 24 '15 at 2:38
  • $\begingroup$ if you worry about the numerical stability, you may try elastic net regularization, i.e. L1-norm and L2-norm simultaneously. $\endgroup$ – Yu Zhang Apr 24 '15 at 2:54
  • $\begingroup$ Well it seems like I'm on the right track then, because that's what I already did. Although I got similar results for mixing parameters of 0.2, 0.5, 0.8, and 1 (i.e. pure LASSO). It's good to know that I don't have to worry about this with the elastic net; do you know any references I can cite to that effect? $\endgroup$ – shadowtalker Apr 24 '15 at 2:57
  • $\begingroup$ sorry, don't have exact paper on this topic. $\endgroup$ – Yu Zhang Apr 24 '15 at 3:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.