# Two sample proportions: How to test the hypothesis that the difference in proportions is less than a specified threshold?

Say I have treatment and control, perfectly randomized.

I have 70000 conversions for treatment in 80000 users. I have 70100 converions for control in 800100 users.

My null hypothesis is that there is at least a one percent difference between the conversion rates.

prop.test does not allow you to set a "difference" level, such as t.test does. Given how many observations I have, it does seem like I could simply use a t.test.

But I wanted to see: How can I use a z-score test that tests for a certain threshold as the null hypothesis?

Should the numerator, instead of $$\mu_1 - \mu_2$$ instead be $$\mu_1 - \mu_2 - c$$, where $$c$$ is the difference I want to test? (say, 0.01 for a 1 point difference in proportions?)

• Just to clarify, do you suspect that there is at least a one percentage point difference in conversion rates (i.e. that treatment is better than control by at least one percentage point?) Sep 17, 2023 at 23:21
• Yes. My null hypothesis is that. @DemetriPananos Sep 18, 2023 at 4:22
• The conversation rate is 10 times that high, so the p value of any test will be zero. The question is: why is such a p value of any interest? Sep 18, 2023 at 6:25
• It's just the operating null hypothesis of the business. @MichaelM Sep 18, 2023 at 7:07
• So: the answer is: the p value is 0 no matter how you calculate it. Sep 18, 2023 at 7:12

You say that your null is that the difference in conversion is at least 1 percentage point, but in the comments you also agree that you suspect the change is at least one percentage point. Those two statements are in conflict, so I will do my best to interpret what you need. If I have misunderstood, just let me know and I'll edit where approprioate.

I think you suspect the change is at least one percentage point, so the corresponding null is actually

$$H0: p_1 - p_0 < 0.01$$

and the alternative is

$$HA: p_1 - p_0 \geq 0.01$$

This is a classical one sided test and we can do this with R in the following way given the numbers you provide


Ntrt <- 80000
ytrt <- 70000

Nctrl <- 800100
yctrl <- 70100

p1 <- ytrt/Ntrt
p0 <- yctrl/Nctrl
phat <- (yctrl + ytrt)/(Ntrt + Nctrl)
vrp <- \(x) x*(1-x)

z <- (p1 - p0 - 0.01)/sqrt( vrp(phat)*(1/Ntrt + 1/Nctrl)  )

# This is essentially 0 given your data
pval <- 1 - pnorm(z)


An easier way to do this is to just use prop.test and determine of 0.01 falls within the one sided confidence interval

prop.test(
c(ytrt, yctrl),
c(Ntrt, Nctrl),
correct = F,
alternative = 'greater'
) -> test

test\$conf.int
[1] 0.7853936 1.0000000



Because the confidence interval only contains differences larger than 0.01, you would reject the null in this case.