Suppose I have $K$ populations each consisting of $n_i$ ($i=1\dots K$) observations $x_{j,i}$ ($j=1 \dots n_i$). Each observation is comming from a Bernoulli distributed random variable $X_{j,i} \sim Bern(p_i)$. I would like to test the hypothesis

$H_0: p_1= \dots p_K$


$H_1:$ "not all $p_i$'s are equal".

For each $i$ I also want to run through all the $j \neq i$ to test if $p_i=p_j$.

I read about the Kruskal Wallis test, which seems to be kind of what I need except that it requires the data to come from a continuous distribution. Does anybody know what tests I can use in this case? I would be very greatful if you could explain how I should use the test as well.

Kind regards,

  • $\begingroup$ I don't see how a Kruskal-Wallis test is appropriate for the kind of data you describe. See my Answer on how to use a chi-squared test. $\endgroup$
    – BruceET
    Sep 2 '19 at 0:43

One possibility is a chi-squared test for homogeneity for the $K$ populations. For each of the $K$ samples you need the number of Successes and Failures out of $n_i$ trials, $i = 1, \dots, K$.

Data. Here is data simulated in R for $K = 5$ with most $p_i$'s different.

x1 = rbinom(1, 100, .4);  x2 = rbinom(1, 150, .5)
x3 = rbinom(1, 110, .6);  x4 = rbinom(1, 120, .6)
x5 = rbinom(1, 150, .7);  succ = c(x1, x2, x3, x4, x5)
fail = c(100,150,110,120,150) - succ
DTA = rbind(succ, fail); DTA

DTA = rbind(succ, fail); DTA
     [,1] [,2] [,3] [,4] [,5]
succ   37   76   67   64   99
fail   63   74   43   56   51

Main chi-squared test. A chi-squared test of homogeneity strongly rejects the null hypothesis that all five $p_i$s are equal with P-value $0.0001 < 0.05.$


        Pearson's Chi-squared test

data:  DTA
X-squared = 23.121, df = 4, p-value = 0.0001198

Additional information. You can use $-notation to look at observed counts (echoing the data), expected counts (based on the null hypothesis), and Pearson residuals (their squares are 'contributions to chi-squared', which add up to the chi-squared statistic 23.121).

  • You should check observed counts to make sure data were correctly entered into the procedure,

  • Expected counts to make sure they are all above 3 and mostly above 5, so that the chi-squared statistic will have approximately a chi-squared distribution with 4 degrees of freedom.

  • A look at Pearson residuals with absolute values above 2 (especially 3) will help to find possibly interesting discrepancies between observed and expected counts.

See related output from R below:

      [,1] [,2] [,3] [,4] [,5]
 succ   37   76   67   64   99
 fail   63   74   43   56   51

          [,1]     [,2]     [,3]     [,4]     [,5]
 succ 54.44444 81.66667 59.88889 65.33333 81.66667
 fail 45.55556 68.33333 50.11111 54.66667 68.33333

           [,1]       [,2]       [,3]       [,4]      [,5]
 succ -2.364179 -0.6270544  0.9188917 -0.1649572  1.918049
 fail  2.584559  0.6855062 -1.0045474  0.1803339 -2.096842

Ad-hoc tests. Because $H_0$ was rejected, it is appropriate to make 'pairwise comparisons' between $p_i$s. Here it is reasonable to compare $p_1$ and $p_5$. This can be done using bracket notation [] to select the two relevant columns of DTA.

[1] 1.18377e-05

Bonferroni criterion. Using the Bonferroni method to prevent 'false discovery' you should declare such an ad hoc comparison as significant only if the P-value is below $0.05/m$ where $m$ is the number of comparisons performed. The ad hoc comparison of $p_1$ with $p_5$ is significant by this criterion.

By contrast, we do not find a significant difference between $p_3$ and $p_4.$

[1] 0.3049811
  • $\begingroup$ Great reply, thanks! $\endgroup$
    – user202542
    Sep 4 '19 at 10:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.