I was working on combining the "rare" categories into a single category/group "others" automatically in a univariate categorical variable.

Suppose I have A, B, C, D, and E categories in some categorical variable in which the distribution goes like [40%, 35%, 22%, 2%, 1%] then I would group D and E as the "other" category.

Here, handpicking a threshold like <0.05 won't work for all cases.

My question is if there is a way to automatically get the threshold or some kind of dynamic threshold for the given distribution to detect the "rare" categories.

I have seen some methods:

  • HBOS (Histogram based outlier score)
  • Chi-square residuals

But still, I am not confident about them. HBOS looks promising but if there is some other appropriate way to do this, I would probably go for that instead.

I also have thought of using IQR and zscore for this, but I am a bit skeptical of using them for the categorical variable.

The reason behind doing this "grouping" is that, in machine learning, many times there are so many distinct categories in a variable, and the model (especially the tree model) gives them so much importance because of that variable's multiple categories having a single occurrence. In such a scenario, we usually group such categories and hope for a model to improve the accuracy.

It is where I am looking for a dynamic threshold to "detect" the rare categories.




Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.