This is a mockup of a dataset I am currently working on:
> df <- data.frame(name = LETTERS[1:3], trials = c(40, 60, 60), successes = c(22, 30, 45)) > df$success.rate = df$successes / df$trials > df name trials successes success.rate 1 A 40 22 0.55 2 B 60 30 0.50 3 C 60 45 0.75
I have a few hundred groups of varied size like this one, and what I want to do is figure out if the success rates varies between individuals.
My first though was to run a chi-squared test for goodness of fit to a uniform distribution, like so:
> with(df, chisq.test(success.rate)) Chi-squared test for given probabilities data: success.rate X-squared = 0.0583, df = 2, p-value = 0.9713 Warning message: In chisq.test(success.rate) : Chi-squared approximation may be incorrect
Which I find problematic since it doesn't take into account the fact that each individual has a different number of trials (not to mention using proportions is problematic in itself here). So my next step was to run it like this:
> with(df, chisq.test(x = successes, p = trials, rescale.p = TRUE)) Chi-squared test for given probabilities data: successes X-squared = 3.3711, df = 2, p-value = 0.1853
Which is an indication that the weighted number of successes within this group is uniformly distributed.
However, intuitively my boss and I see a relevant difference between individuals B and C. Both have the same number of trials and yet one has had success in 15 more cases than the other. We believe that difference is unlikely due to chance and should therefore show up on some statistic.
I believe that the chi-squared test conducted above answers a different question. After all, it's one thing to test if the heights of a group of people are normally distributed, and a completely different thing to say that the guy measuring 1,80 m is as tall as the guy with 1,60 m.
As alternatives to detecting the difference in performance of those individuals, we have already considered and tested many things from simple range checking (we would raise red flags on groups where the range of success rates was higher than 0.20) to crude confidence interval setting ($Q1 + 1.5 I\!Q\!R$, $\mu \pm 1.96 \sigma$ and other combinations thereof). I've also tried testing for other distributions (e.g., Normal through
shapiro.test(df$success.rate)), but since my samples are usually so small it's no wonder I'll almost always get a high p-value.
The amount of groups we're analyzing and the diversity of sizes (which can range from 2 to 15) has made us look towards a non-graphic and non-parametric solution, but I am unsure about which would be the most appropriate.