0
$\begingroup$

Let's say we are evaluating two algorithms.

 Algorithm 1 gets p1 = 80% conversion
 Algorithm 2 gets p2 = 85% conversion 

The two sample sizes are

 n1 and n2 

respectively.

I'm trying to test the following hypothesis

$H_0:$ p1 = p2 $H_1:$ p1 $\neq$ p2

I though when the sample is large

$p1-p2$ ~ $N(p1-p2,\ \frac{p1(1-p1)}{n1} + \frac{p2(1-p2)}{n2})$

so the test statistic is

$z = \frac{p1-p2}{\sqrt{\frac{p1(1-p1)}{n1} + \frac{p2(1-p2)}{n2}}}$

But then I read in multiple places [http://mlwiki.org/index.php/Binomial_Proportion_Tests] that the two-sample proportions actually has a test statistic of

$z = \frac{p1-p2}{\sqrt{p(1-p)[\frac{1}{n1} + \frac{1}{n2}]}}$

where $p$ is the pooled proportion

$p = \frac{x1+x2}{n1+n2}$

where $x1,\ x2$ are the number of conversions.

What is the difference between the two? And when should I use each of them?

$\endgroup$
  • $\begingroup$ The test is conducted using the distribution of the test statistic assuming that $H_0$ is true. If $H_0$ is true then $p_1 = p_2$ and that provides a rationale for using the pooled proportion. // I tend to use the first version for one-sided tests and when $n_1$ and $n_2$ are very different, but I can't defend that choice. // In a recent edition, one commonly used textbook has changed from the pooled proportion to use the first version you mention. // Recent releases of Minitab offer a choice. // And then there's Fishers 'Exact Test'. Maybe someone here has a sound argument for a preference. $\endgroup$ – BruceET Jul 5 '18 at 6:54
2
$\begingroup$

Both test statistics are used in practice.

What is the difference between the two?

The difference is in the assumptions. When using unpooled method the following are the assumptions:

a) The two samples are independent and are distributed approximately normal.

When using pooled method following are the assumptions:

a) The two samples are independent and are distributed approximately normal.

b) population variances are equal

And when should I use each of them?

This depends on the choice of the hypothesis. When testing "no difference" between the two proportions then variance should be pooled. This is because under H0 we can approximate both the proportions by the pooled proportion. So we should go with this statistic:

$z = \frac{p1-p2}{\sqrt{p(1-p)[\frac{1}{n1} + \frac{1}{n2}]}}$

In the example you mention, Ho says "no difference". So, pooled should be used.

When testing for the difference between two proportions, e.g. the difference of 0.1, which implies that Ho is a number other than 0 then unpooled should be used. This is because under null hypothesis the proportions are different so pooled proportion should not be used to approximate the two proportions.

Put simply, pooled method is used when the null hypothesis is that the two proportions are equal.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.