I am working with 82 binary features constructed from six categorical features. I have about 1,600 observations. Some of these features correspond to extremely rare categories. Some of them have only one or two 1's in the entire sample; others have in the teens, and a few others have in the 20s and 30s.
There is a lot of natural variation in the data, so these very rare 1's are not likely to be informative. But how rare is "too rare?" The categories that appear only twice in 1,600 observations are probably too rare. But is 20 appearances still too rare? What principled approach can I take to choosing a rarity cutoff?
This also begs the question of what to do with the categories that are "too rare." I'm tempted to drop them. But given that they correspond to mutually exclusive blocks of indicators, that means I am effectively rolling the rare categories in with the baseline category. This isn't such a big deal for one or two observations, but I'm concerned about doing it with too many features at once.
Edit: I'm not just using these features in a predictive model. I'm also interested in the associations between features. I'm going to use either correlation or Cramer's V; I'd like to just drop a baseline category and compute the correlation matrix, but I imagine there are problems with that approach (and it's a whole separate question anyway).
