When can we bucket groups together based on groups success rate? I have a data set with a nominal data feature that has 20 levels. I want to label those levels in a way that I can say "Level A is Good", "Level B is Medium"... The data set is made of samples with a binary feature (0, 1) which indicates a success rate.
So far, I've computed the average success rate for each of my levels and used percentiles and assigned a label based on the average success rate for each level. My issue is that some of my levels have a very low count. The level with the highest row count has 14704 while the lowest has 1 row.
My question is, is there a better way to categorize my data that takes into consideration the low count of some of my category's levels. Would it be wise to merge low count levels into one group "Other" and if not, could you explain me why this would not be wise?
Thank you in advance for your precious time and help!
 A: Let $i\in[1..20]$ be the levels of your nominal feature.
Let $P_i$ be the population having feature $i$. Let $S_i = \{x_{i,j}|j\in[1..n_i]\}$ be your sample of size $n_i$ of $P_i$. Let $B(p_i)$ the bernoulli trial for each observation of $P_i$ defined as success. You're estimating $p_i$ with the maximum likelihood estimator $\hat{p_i}$ = $\frac{1}{n}\sum_{j=0}^{n_i}x_{i,j}$ based on $S_i$, which you want to bucket in groups $g\in[1..3]$ correctly. $y_i=\sum_{j=0}^{n_i}x_{i,j}$ is the number of successes of sample $S_i$ which is a realisation of the random variable $Y_i\sim\text{Bin}(n_i,p_i)$
Let $q_i = 1-p_i$, $\mu_i=n_i p_i, \sigma_i=\sqrt{n_ip_iq_i}$. You can use the normal approximation $Y_i\sim\mathcal{N}(\mu_i, \sigma_i²)$ if $n_i\cdot p_i > 5$ and $n_i\cdot q_i > 5$.
Basically, you will be confident in your estimator only if $n_i > \frac{5}{p_i}$ and $n_i > \frac{5}{q_i}$. Your group with only 1 observation provides you practically no information on its population. So before grouping populations on $p_i$, I would either get more observations of the populations with weak estimates of $p_i$; group these populations according to another attribute; or group these low sample populations together until you have a decent estimate of the probability of success of the new "Other" population.
