Comparing statistically the distribution of frequency between several groups

Question

I have a set of data corresponding to several groups (group1, group2,...) split into 7 categories (cat1, cat2...). For each group, I have counted the number of individuals belonging to each category. So all the variables I am dealing with are discrete. In the end, I have tables that look like that :

Cat/group	cat1	cat2	cat3
group1	3	0	5
group2	5	18	0
group3	0	81	0

So, the sample is very unequally distributed since I can have categories with 0 individuals and others with 81.

What I would like to do is statistically compare the distribution, to argue that one group described by its effectiveness in each category is different or not from another. Until now, I have tried Mann-Witney, the Fisher test on contingency table and the K-S test. However, I am not convinced I am using the right tests. Indeed, K-S is not recommended for discrete data, Fisher seems very restricted and I am not sure Mann-Witney is adequate with very unequally distributed samples. I would very much appreciate some help here. Thank you!

Answer 1

The Mann-Whitey and the Kolmogorov-Smirnov test are for continuous data so they are not applicable here.

Since you do not seem to have repeated measures a chi-squared test will fill the bill. The chi-squared test aims to assess the null hypothesis of the lack of association between the two categorical variables. Rejecting the null hypothesis means that the two variables are dependent thus the conditional distributions of say group category given the group are necessarily statistically different.

You can perform this in R by running

mm <- matrix(c(3,0,5,5,18,0,0,81,0), byrow = TRUE, ncol=3)
ch <- chisq.test(mm)

Warning message:
In chisq.test(mm) : Chi-squared approximation may be incorrect


> ch

    Pearson's Chi-squared test

data:  mm
X-squared = 96.541, df = 4, p-value < 2.2e-16

The Warning you see is because of the low expected frequency (i.e. expected frequencies <5 ). Indeed, you can check it yourself

> ch$expected
          [,1]      [,2]      [,3]
[1,] 0.5714286  7.071429 0.3571429
[2,] 1.6428571 20.330357 1.0267857
[3,] 5.7857143 71.598214 3.6160714

The problem is that the asymptotic $\chi^2$ distribution may be far from valid so the associated p-value may not be accurate.

Instead of relying on the above asymptotic distribution, a workaround to this issue is to compute the p-value via simulation. In R you can compute this p-value as follows

> chisq.test(mm, simulate.p.value = TRUE, B=10^4)

    Pearson's Chi-squared test with simulated p-value (based on
    10000 replicates)

data:  mm
X-squared = 96.541, df = NA, p-value = 9.999e-05

In both cases, the p-value is extremely small so we can reject the null and conclude that the two variables are statistically associated.

As an alternative, you can use the Fisher's exact test, especially if the sample size is very small. Be aware that this test test assumes that the marginal distributions of your table are fixed, which may not necessarily be true in your case. In R you can run this test by the command fisher.test. Some people argue that Fisher's exact test tends to be conservative. However, with sufficiently large sample sizes, the chi-squared and Fisher's exact tests will lead to the same decision.