I have $3$ groups ($n_1 = 30$, $n_2 = 20$, $n_3 = 5$), and I am looking at anxiety scores, which is binary (0 = no anxiety, 1 = anxiety present). I want to perform pairwise comparisons i.e is there a significant difference in anxiety scores between group 1 v.s group 2? Group 2 v.s group 3? Group 1 v.s group 3?

I am not quite sure if t-test would be appropriate here since I don't know if the variables are iid normal, so I am thinking of using the Wilcoxon signed-rank test (wilcox.test(..., paired = FALSE) in R). What other test would be appropriate here?


The answer would probably include repeating this post dealing with the resilience of the chi square test even when expected cell counts are below 5. So a chi square test may be the answer, and it could be approached as a goodness of fit (GOF), comparing the distribution of the frequency of "anxious" subjects to a uniform distribution across the 3 groups.

However, there will be probably a warning message when running the chi square test. I simulated the data set (the code is here, and also pasted below this paragraph for convenience. There were the same number of subjects as in the original post group a = 30, b =20 and c = 5:

subjects <- c(group_a <- rep("a", 30), group_b <- rep("b", 20), group_c <- rep("c", 5))
sam <-      c(sample(c(c(rep("N",25)), c(rep("A",5)))),
            sample(c(c(rep("N",10)), c(rep("A",10)))),
            c(sample(c("A","N"), 5, replace = T)))
(tab = table(subjects,sam))  
subjects  A  N
       a  5 25
       b 10 10
       c  3  2

The group a was set up as much less anxious: 25 N for "no anxiety".

The test output was:

    (p_observed = chisq.test(tab, correct = F))
    Pearson's Chi-squared test

data:  tab
X-squared = 7.9142, df = 2, 

$p-value = \color{orange}{0.01912}$

But there was a warning message:

Warning message:
In chisq.test(tab, correct = F) :
  Chi-squared approximation may be incorrect

So I wanted to see if we could come up with an ad hoc permutation test, and shuffle the A, N labels across all subjects, under the null hypothesis of no differences in distribution of anxious subjects across groups, and then run a chi square test on the tabulated resulting frequencies. Disregarding thus any issues with the small size of group 3, and just paying attention to the relative p value of every iteration with respect to the others, I think it is fair to say that the proportion of permutations with a p value lower than the observed in the actual data is an exact p value - a simulated simulated Fisher test, or, probably more exactly, a permutation test.

Here's the code in R, and the results:

            (tab = table(subjects,sam))  
            (p_observed = chisq.test(tab, correct = F))
chisq.test(tab, simulate.p.value = T)

pval <- c(NA, length(sam))

            for (i in 1:1e4){
       anx <- sample(sam, length(sam), replace = F)
       tab     <- table(subjects,anx)
       pval[i] <- chisq.test(tab, correct = F)$p.value
(p_value = mean(pval < p_observed$p.value))

The p_value = $\color{red}{0.0186}$ was lower than the initially calculated with chisq.test.

I was surprised that this p value was also lower than the calculation via Monte Carlo simulation within the R built-in function:

chisq.test(tab, simulate.p.value = T)

    Pearson's Chi-squared test with simulated p-value (based
    on 2000 replicates)

data:  tab
X-squared = 7.9142, df = NA, p-value = 0.02099

After getting a significant result, pairwise comparisons can be performed with Bonferroni correction (level of significance $0.05 /\text{no.hypotheses} = 0.05 / 3 = \color{blue}{0.0167}$). These can be obtained directly with the R built-in fisher.test function:

Between group a and b:

fisher.test(tab[1:2,], alternative = "two.sided")
p-value = 0.02546

Between b and c:

fisher.test(tab[2:3,], alternative = "two.sided")
p-value = 1

And a and c:

fisher.test(tab[c(1,3),], alternative = "two.sided")
p-value = 0.06654

Oddly, none of the results is significant, because of the conservative nature of the Fisher test. If we were to run a chisq.test between a and b - and the size of the samples would clearly allow it - we would get a statistically significant result:

$p-value = \color{green}{0.01174}.$


Are you trying to compare proportions between 2 groups? If yes I think you can use Z-test where Z= (p1-p2)/sqrt(p(1-p)/(1/n1+1/n2)) and p is the proportion when 2 groups are combined.

  • $\begingroup$ Welcome to our site! How would you propose coping with the two salient complications: namely, (1) that one of the groups has only five elements, suggesting a Z test could be a poor approximation; and (2) that three interdependent comparisons will be made, indicating the need somehow to adjust the p-values of the separate test results? $\endgroup$ – whuber Jan 29 '16 at 14:39
  • 1
    $\begingroup$ @whuber I was curious to see if the conversation got going further, but since it may not be the case... I think the answer is a GOF chi square to test whether the proportion of anxiety cases are distributed uniformly across groups. The problem with the low counts in group 3 may not be prohibitive. Pairwise comparisons with Bonferroni or other adjustments may follow. Any role for Freeman test, or any R Monte Carlo to permute tables? $\endgroup$ – Antoni Parellada Jan 29 '16 at 20:43
  • 1
    $\begingroup$ @Antoni All sound like good ideas. You could add to them other well-known approaches such as logistic regression. Whether or not the low count in a group is a problem can be decided in various ways, but what is of greatest importance in this forum is to recognize that it is a potential problem, rather than to ignore it. $\endgroup$ – whuber Jan 29 '16 at 20:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.