How to get the dynamic threshold to detect rare categories from the univariate categorical variable?

Question

I was working on combining the "rare" categories into a single category/group "others" automatically in a univariate categorical variable.

Suppose I have A, B, C, D, and E categories in some categorical variable in which the distribution goes like [40%, 35%, 22%, 2%, 1%] then I would group D and E as the "other" category.

Here, handpicking a threshold like <0.05 won't work for all cases.

My question is if there is a way to automatically get the threshold or some kind of dynamic threshold for the given distribution to detect the "rare" categories.

I have seen some methods:

• HBOS (Histogram based outlier score)
• Chi-square residuals

But still, I am not confident about them. HBOS looks promising but if there is some other appropriate way to do this, I would probably go for that instead.

I also have thought of using IQR and zscore for this, but I am a bit skeptical of using them for the categorical variable.

The reason behind doing this "grouping" is that, in machine learning, many times there are so many distinct categories in a variable, and the model (especially the tree model) gives them so much importance because of that variable's multiple categories having a single occurrence. In such a scenario, we usually group such categories and hope for a model to improve the accuracy.

It is where I am looking for a dynamic threshold to "detect" the rare categories.

Thanks.