How do I Calculate the correct sample size for a test of difference in proportion in this specific case?

Question

My brother is comparing a number of advertisement campaings. He wants to know how large the samples must be in order to determine whether a 2 percent difference between twp proportions is going to be significant.

He assures me that he will organize the campaings so that the sample sizes are equal. He told me to use success of 5 and 7 percent for the calculation.

I've tried to work this out myself, as follows:

In order to calculate the pooled variance we first need the pooled proportion, so:

$$p=\frac{0.05+0.07}{n_1+n_2}=\frac{0.12}{2n_1}$$

The expression for the pooled variance is:

$$\sigma=\frac{p(1-p)}{n_1}+\frac{p(1-p)}{n_2}$$

But since $n_1=n_2$ we get:

$$\sigma=\frac{2p(1-p)}{n_1}$$

We substitute $p$

$$\sigma=\frac{2\frac{0.12}{2n_1}(1-\frac{0.12}{2n_1})}{n_1}$$

$$\sigma=\frac{2(\frac{0.12}{2n_1}-(\frac{0.12}{2n_1})^2)}{n_1}=\frac{(\frac{0.24}{2n_1}-(\frac{2(0.12)^2}{4n_1^2}))}{n_1}=\frac{0.24}{2n_1^2}-(\frac{2(0.12)^2}{4n_1^3}))$$

We want to simplify this expression and gain a common denomitator so:

$$\frac{0.24(4n_1^3)}{2n_1^2(4n_1^3)}-\frac{2(0.12)^2(2n_1^2)}{4n_1^3(2n_1^2)}=\frac{0.24(2n_1)-2(0.12)^2}{4n_1^3}=\sigma^2$$

$$\sigma=\sqrt{\frac{0.24(2n_1)-2(0.12)^2}{4n_1^3}}$$

Now in order to calculate or z score, we write:

$$z=\frac{p_1-p_2}{\sigma}=1.96$$

$$=>\frac{0.02}{\sigma}=1.96$$

$$=>\frac{0.02}{1.96}=\sigma$$

$$=>(\frac{0.02}{1.96})^2=\sigma^2$$

$$=>(\frac{0.02}{1.96})^2=\frac{0.24(2n_1)-2(0.12)^2}{4n_1^3}$$

$$=>(\frac{0.02}{1.96})^2(4n_1^3)=0.24(2n_1)-2(0.12)^2$$

$$=>(\frac{0.02}{1.96})^2(4n_1^3)-0.24(2n_1)=-2(0.12)^2$$

$$=>(4n_1^3)-\frac{0.24(2n_1)}{(\frac{0.02}{1.96})^2}=-\frac{2(0.12)^2}{(\frac{0.02}{1.96})^2}$$

$$=>64(n_1^3)-4609.92n_1=276.5952$$

Questions:

1. Is this even the correct answer? I would really prefer not having to solve a cubic equation every time I need to figure out a simple sample size...

2. Given that this actually is correct, then how on earth do I solve for $n_1$?

Answer 1

In order to find sample size, in addition to the difference in proportions (given as .02), you need significance level (I suppose 5%) and desired power (let's say 80% or 90%). You do not state a target power, nor give enough explanation along with your computations to make your strategy clear (not to me, anyhow). So I can't say how to simplify your approach.

Using a recent release of Minitab, we get the following output, showing that you need sample sizes of between 2200 and 3000 in each group. [Most statistical software has 'power and sample size' procedures, and there are on-line sample size calculators (of varying accuracy and ease of use).]

Power and Sample Size 

Test for Two Proportions

Testing comparison p = baseline p (versus ≠)
Calculating power for baseline p = 0.05
α = 0.05

              Sample  Target
Comparison p    Size   Power  Actual Power
        0.07    2213     0.8      0.800142
        0.07    2962     0.9      0.900095

The sample size is for each group.

If you can guess the approximate value of $n$ (maybe roughly as $n \approx 1/\delta^2,$ where $\delta = .02$ here), then you can do a simple simulation in R to see the resulting power. [In R the test of two binomial proportions is prop.test.]

set.seed(2021)
n = 2700
pv = replicate(10^4, 
              prop.test(c(rbinom(1,n,.05),rbinom(1,n,.07)), c(n,n))$p.val)
mean(pv <= .05)
[1] 0.8608     # power

Answer 2

This will probably be downvoted but according to my current understanding, this is the correct answer.

To put it simply, in the very first formula I assumed that: $$p=\frac{p_1+p_2}{n_1+n_2}$$

This is incorrect, and it should read:

$$p=\frac{x_1+x_2}{n_1+n_2}$$

...Where $x_1$ denotes the number of "successes" in $n_1$ and vice versa.

Since we know that $\frac{x_1}{n2}=0.05$ and that $\frac{x_2}{n2}=0.05$ it must follow that $\frac{x_1+x_2}{n_1}=0.12$ if the samples are equal. The expression for the common proportion is almost equal, apart from the fact that the demoninator is twice as big:

$$\frac{x_1+x_2}{2n_1}$$

Which implies that:

$$\frac{x_1+x_2}{2n_1}=0.06$$

In which case the next steps can be significantly simplified:

$$S^2=\frac{0.06(1-0.06)}{2n_1}$$

If we skip a few steps, this means that:

$$(\frac{0.02}{1.96})^2=\frac{0.06(1-0.06)}{2n_1}$$

$$n_1=\frac{0.06(1-0.06)}{2(\frac{0.02}{1.96})^2}$$

....Which adds up to an $n_1$ of 270.8 or 271 rounded up.