# How to test if 2 binomial samples are equivalent?

Problem:

Let $$X_1,..., X_{n_1} \stackrel{iid}{\sim} \text{Binomial}(n, p_1)$$ and $$Y_1,..., Y_{n_2} \stackrel{iid}{\sim} \text{Binomial}(n, p_2)$$. I want to test the following hypotheses:

$$H_0: p_1 = p_2 \\ H_A: p_1 \neq p_2$$

My approach so far:

Let $$\hat{p}_1 = \frac{ \sum_{i=1}^{n_1} X_i/n }{n_1}$$ and $$\hat{p}_2 = \frac{ \sum_{i=1}^{n_2} Y_i/n }{n_2}$$. Test if these proportions are equal using a chi-squared test of homogeneity (see here for details).

My questions:

1. Is there a better way to do this test? Letting $$p_1$$ and $$p_2$$ be the averages of all individual proportions feels silly because it ignores the distributions of the proportions.

2. Could I let $$\hat{p}_1 = (X_1/n, ..., X_{n_1}/n)$$ and $$\hat{p}_2 = (Y_1/n, ..., Y_{n_2}/n)$$ and use a non-parametric ANOVA (like the Kruskall-Wallis test) to test if the vectors of proportions are different? Would this approach be better than my initial proposal?

• – whuber
Jun 3, 2019 at 19:28
• Jun 3, 2019 at 19:57
• $n_1$ and $n_2$ are the number of samples from each population respectively and $n$ is the number of trials. None of these numbers are necessarily equal to each other. Jun 3, 2019 at 20:13
• Consider this scenario: You have two methods A & B of teaching how to do a task: $n_1 =50$ subjects are taught using A, $n_2=60$ using B. At the end everybody takes an $n=100$ question True/False test. The first subject taught using A gets $X_1 = 83$ of $n=100$ questions right. That person's score is the binomial proportion $83/100 = .83.$ Altogether in the two groups, you have $n_1+n_2 = 50+60 = 110$ of these binomial proportion scores. Is that anything like your design? Please answer Yes or No, and discuss for clarity. If Y, are there several equiv. versions of the test? Or all exactly same? Jun 4, 2019 at 1:02
• OK. Suggest you make details of your Q more exact, as far as possible. About how many questions on exam. Are some quest on the test more important or quite diff than others? Approx values of $n_1$ and $n_2?$ 15? 500? Give a clue what A and B are and how they differ. For your work, what do $p_1$ and $p_2$ actually represent and what would it mean to you if they are significantly different? // If someone wants to answer your Q, they will read your explanation. Don't depend om them reading my long Comment. // If its going to be a chi-sq test, data in table must be counts not proportions. Jun 4, 2019 at 17:12

Under your assumptions, $$X =\sum_1^{n_1} X_i \sim \mathcal{Binom}(n\cdot n_1, p_1) \\ Y = \sum_1^{n_2} Y_i \sim \mathcal{Binom}(n \cdot n_2, p_2)$$ which reduces this to a test of equality of two binomial proportions. Many questions on this site about that. You say:
Letting $$𝑝_1$$ and $$𝑝_2$$ be the averages of all individual proportions feels silly because it ignores the distributions of the proportions.
Maybe it feels so, but under your assumptions, this is a sufficient reduction of the data ... so if it feels silly, maybe because you wonder if the individual probabilities of the $$X_i$$'s and of the $$Y_i$$'s might not be all equal?
Your question 2: Again, under the stated assumptions, this would not be better, as the test would be based on variation (in the individual $$X_i/n, Y_j/n$$ which under the stated model is irrelevant.