I was working on combining the "rare" categories into a single category/group "others" automatically in a univariate categorical variable.
Suppose I have A, B, C, D, and E categories in some categorical variable in which the distribution goes like [40%, 35%, 22%, 2%, 1%] then I would group D and E as the "other" category.
Here, handpicking a threshold like
<0.05won't work for all cases.
My question is if there is a way to automatically get the threshold or some kind of dynamic threshold for the given distribution to detect the "rare" categories.
I have seen some methods:
- HBOS (Histogram based outlier score)
- Chi-square residuals
But still, I am not confident about them. HBOS looks promising but if there is some other appropriate way to do this, I would probably go for that instead.
I also have thought of using
zscore for this, but I am a bit skeptical of using them for the categorical variable.
The reason behind doing this "grouping" is that, in machine learning, many times there are so many distinct categories in a variable, and the model (especially the tree model) gives them so much importance because of that variable's multiple categories having a single occurrence. In such a scenario, we usually group such categories and hope for a model to improve the accuracy.
It is where I am looking for a dynamic threshold to "detect" the rare categories.