There are two categorical variables with four levels, so ordinal variables. Specifically, they are responders and non-responders with heart function that is graded from mild, mild-moderate, moderate, severe. So each patient will either be a responder or non-responder and heart function would fit into one of four grades. I am looking to see if responders or non-responders are more likely to have better heart function. This is complicated by small n-values, which range from 0 - 5 per category. Thanks for any insight!


1 Answer 1


If I understand your explanation correctly, you may have 25 responders and 22 non-responders, possibly with different distributions of heart-function scores 1 through 4 between the two groups. Here are fake data sampled in R, to provide an example:


res = sample(1:4, 25, rep=T, p=c(1,2,3,4))
[1] 1 2 2 2 3 3 3 3 3 3 3 3 3 3 4 4 4 4 4 4 4 4 4 4 4
t.r = tabulate(res)
[1]  1  3 10 11

non = sample(1:4, 22, rep=T, p=c(4,3,2,1))
[1] 1 1 1 1 1 1 1 1 1 1 1 1 2 2 2 2 3 3 3 4 4 4
t.n = tabulate(non);  t.n
[1] 12  4  3  3

TAB = rbind(t.r, t.n); TAB
    [,1] [,2] [,3] [,4]
t.r    1    3   10   11
t.n   12    4    3    3

The resulting contingency table has some small observed counts, which result is less-than-ideal expected counts in a chi-squared test.


        Pearson's Chi-squared test

data:  TAB
X-squared = 17.672, df = 3, p-value = 0.000514

Warning message:
In chisq.test(TAB) : 
  Chi-squared approximation may be incorrect

Expected counts below $5$ may lead to incorrect P-values. (Many statisticians would say that a few expected counts of at least 3 amongst counts predominantly above 5 would be OK.)

        [,1]     [,2]     [,3]     [,4]
t.r 6.914894 3.723404 6.914894 7.446809
t.n 6.085106 3.276596 6.085106 6.553191

However in R, using chisq.test with parameter sim=T, it is often possible to simulate a reliable P-value for this test, even when some expected counts are too low. For our fake data, the P-value is sufficiently small to reject the null hypothesis that responders and non-responders have the same distributions of scores.

chisq.test(TAB, sim=T)

        Pearson's Chi-squared test 
        with simulated p-value 
        (based on 2000 replicates)

data:  TAB
X-squared = 17.672, df = NA, p-value = 0.0004998

Ordinarily, I would not suggest a two-sample Wilcoxon rank sum test for two reasons: (a) many ties in the data; (b) markedly different shapes of distributions for responders and non-responders. However, the Wilcoxon RS test does find a significant difference.

wilcox.test(res, non)

        Wilcoxon rank sum test 
        with continuity correction

data:  res and non
W = 448.5, p-value = 0.0001292
alternative hypothesis: 
  true location shift is not equal to 0
Warning message:
In wilcox.test.default(res, non) : 
  cannot compute exact p-value with ties

Moreover, empirical CDF (ECDF) plots of the two distributions show that the distribution of responders stochastically dominates (tends to have larger values than) the distribution of non-responders. The former distribution (blue) lies to the right (hence also below) the latter (brown).

enter image description here

There may be occasions where the Wilcoxon test or the ECDF plots may help to explain a difference in distributions found by the chi-squared test.

  • $\begingroup$ Thank you for this thorough examination of my problem. My sample size is even lower than this. There are 5 responders and 7 non-responders. How would a Wilcoxon test be able to be used here? I thought the Wilcoxon test was for comparing means between non-parametric groups? Separately, I realize now I could assign each category of heart function a number. For example, assigning a number of 1 through 4 for ascending heart function. This would instead give a mean for each patient group and then allow for a non-parametric comparison test. $\endgroup$ Feb 1, 2021 at 14:17
  • $\begingroup$ I'm not sure you have enough observations for a Wilcoxon test to be useful. $\endgroup$
    – BruceET
    Feb 1, 2021 at 18:22
  • $\begingroup$ For binary Y the minimum sample size needed to only estimate the intercept is n=96. That's before you add regression coefficients. This situation is futile. $\endgroup$ Sep 22, 2023 at 11:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.