Nonparametric multiplecomparison tests for discrete data

Suppose I have $$K$$ populations each consisting of $$n_i$$ ($$i=1\dots K$$) observations $$x_{j,i}$$ ($$j=1 \dots n_i$$). Each observation is comming from a Bernoulli distributed random variable $$X_{j,i} \sim Bern(p_i)$$. I would like to test the hypothesis

$$H_0: p_1= \dots p_K$$

against

$$H_1:$$ "not all $$p_i$$'s are equal".

For each $$i$$ I also want to run through all the $$j \neq i$$ to test if $$p_i=p_j$$.

I read about the Kruskal Wallis test, which seems to be kind of what I need except that it requires the data to come from a continuous distribution. Does anybody know what tests I can use in this case? I would be very greatful if you could explain how I should use the test as well.

Kind regards,

• I don't see how a Kruskal-Wallis test is appropriate for the kind of data you describe. See my Answer on how to use a chi-squared test. Sep 2 '19 at 0:43

One possibility is a chi-squared test for homogeneity for the $$K$$ populations. For each of the $$K$$ samples you need the number of Successes and Failures out of $$n_i$$ trials, $$i = 1, \dots, K$$.

Data. Here is data simulated in R for $$K = 5$$ with most $$p_i$$'s different.

set.seed(901)
x1 = rbinom(1, 100, .4);  x2 = rbinom(1, 150, .5)
x3 = rbinom(1, 110, .6);  x4 = rbinom(1, 120, .6)
x5 = rbinom(1, 150, .7);  succ = c(x1, x2, x3, x4, x5)
fail = c(100,150,110,120,150) - succ
DTA = rbind(succ, fail); DTA

DTA = rbind(succ, fail); DTA
[,1] [,2] [,3] [,4] [,5]
succ   37   76   67   64   99
fail   63   74   43   56   51


Main chi-squared test. A chi-squared test of homogeneity strongly rejects the null hypothesis that all five $$p_i$$s are equal with P-value $$0.0001 < 0.05.$$

chisq.test(DTA)

Pearson's Chi-squared test

data:  DTA
X-squared = 23.121, df = 4, p-value = 0.0001198


Additional information. You can use $-notation to look at observed counts (echoing the data), expected counts (based on the null hypothesis), and Pearson residuals (their squares are 'contributions to chi-squared', which add up to the chi-squared statistic 23.121). • You should check observed counts to make sure data were correctly entered into the procedure, • Expected counts to make sure they are all above 3 and mostly above 5, so that the chi-squared statistic will have approximately a chi-squared distribution with 4 degrees of freedom. • A look at Pearson residuals with absolute values above 2 (especially 3) will help to find possibly interesting discrepancies between observed and expected counts. See related output from R below:  chisq.test(DTA)$obs
[,1] [,2] [,3] [,4] [,5]
succ   37   76   67   64   99
fail   63   74   43   56   51

chisq.test(DTA)$exp [,1] [,2] [,3] [,4] [,5] succ 54.44444 81.66667 59.88889 65.33333 81.66667 fail 45.55556 68.33333 50.11111 54.66667 68.33333 chisq.test(DTA)$res
[,1]       [,2]       [,3]       [,4]      [,5]
succ -2.364179 -0.6270544  0.9188917 -0.1649572  1.918049
fail  2.584559  0.6855062 -1.0045474  0.1803339 -2.096842


Ad-hoc tests. Because $$H_0$$ was rejected, it is appropriate to make 'pairwise comparisons' between $$p_i$$s. Here it is reasonable to compare $$p_1$$ and $$p_5$$. This can be done using bracket notation [] to select the two relevant columns of DTA.

chisq.test(DTA[,c(1,5)])$p.val [1] 1.18377e-05  Bonferroni criterion. Using the Bonferroni method to prevent 'false discovery' you should declare such an ad hoc comparison as significant only if the P-value is below $$0.05/m$$ where $$m$$ is the number of comparisons performed. The ad hoc comparison of $$p_1$$ with $$p_5$$ is significant by this criterion. By contrast, we do not find a significant difference between $$p_3$$ and $$p_4.$$ chisq.test(DTA[,c(3,4)])$p.val
[1] 0.3049811

• Great reply, thanks! Sep 4 '19 at 10:10