# Detecting heterogeneity in groups

This is a mockup of a dataset I am currently working on:

> df <- data.frame(name      = LETTERS[1:3],
trials    = c(40, 60, 60),
successes = c(22, 30, 45))
> df$success.rate = df$successes / df$trials > df name trials successes success.rate 1 A 40 22 0.55 2 B 60 30 0.50 3 C 60 45 0.75  I have a few hundred groups of varied size like this one, and what I want to do is figure out if the success rates varies between individuals. My first though was to run a chi-squared test for goodness of fit to a uniform distribution, like so: > with(df, chisq.test(success.rate)) Chi-squared test for given probabilities data: success.rate X-squared = 0.0583, df = 2, p-value = 0.9713 Warning message: In chisq.test(success.rate) : Chi-squared approximation may be incorrect  Which I find problematic since it doesn't take into account the fact that each individual has a different number of trials (not to mention using proportions is problematic in itself here). So my next step was to run it like this: > with(df, chisq.test(x = successes, p = trials, rescale.p = TRUE)) Chi-squared test for given probabilities data: successes X-squared = 3.3711, df = 2, p-value = 0.1853  Which is an indication that the weighted number of successes within this group is uniformly distributed. However, intuitively my boss and I see a relevant difference between individuals B and C. Both have the same number of trials and yet one has had success in 15 more cases than the other. We believe that difference is unlikely due to chance and should therefore show up on some statistic. I believe that the chi-squared test conducted above answers a different question. After all, it's one thing to test if the heights of a group of people are normally distributed, and a completely different thing to say that the guy measuring 1,80 m is as tall as the guy with 1,60 m. As alternatives to detecting the difference in performance of those individuals, we have already considered and tested many things from simple range checking (we would raise red flags on groups where the range of success rates was higher than 0.20) to crude confidence interval setting ($Q1 + 1.5 I\!Q\!R$,$\mu \pm 1.96 \sigma$and other combinations thereof). I've also tried testing for other distributions (e.g., Normal through shapiro.test(df$success.rate)), but since my samples are usually so small it's no wonder I'll almost always get a high p-value.

The amount of groups we're analyzing and the diversity of sizes (which can range from 2 to 15) has made us look towards a non-graphic and non-parametric solution, but I am unsure about which would be the most appropriate.

I would try something completely different, a logistic mixed effects model. In R that can be estimated by lme4::glmer, search this site for examples, there are plenty. You seem to have no covariates, so that would be a random intercepts model.
Another, maybe more direct way: If all the success probabilities where equal, you had binomial data with the same $$p$$. If the $$p$$'s varies, that looks like overdispersion. A simple model for overdispersed binomial data is the betabinomial, search this site, one famous example can be found here: What are some good exploratory analysis and diagnostic plots for count data?. So you can just fit a betabinomial model and see if it fits much better than a binomial. Since you have many groups, you could go further and estimate some mixture of binomials with a (partially) unknown mixture distribution (in the beta-binomial, the mixture distribution is the beta distribution.) Some papers: Finite mixture models for proportions and A full likelihood procedure for analysing exchangeable binary data.