1
$\begingroup$

Say I have treatment and control, perfectly randomized.

I have 70000 conversions for treatment in 80000 users. I have 70100 converions for control in 800100 users.

My null hypothesis is that there is at least a one percent difference between the conversion rates.

prop.test does not allow you to set a "difference" level, such as t.test does. Given how many observations I have, it does seem like I could simply use a t.test.

But I wanted to see: How can I use a z-score test that tests for a certain threshold as the null hypothesis?

Should the numerator, instead of $\mu_1 - \mu_2$ instead be $\mu_1 - \mu_2 - c$, where $c$ is the difference I want to test? (say, 0.01 for a 1 point difference in proportions?)

$\endgroup$
6
  • $\begingroup$ Just to clarify, do you suspect that there is at least a one percentage point difference in conversion rates (i.e. that treatment is better than control by at least one percentage point?) $\endgroup$ Sep 17, 2023 at 23:21
  • $\begingroup$ Yes. My null hypothesis is that. @DemetriPananos $\endgroup$ Sep 18, 2023 at 4:22
  • $\begingroup$ The conversation rate is 10 times that high, so the p value of any test will be zero. The question is: why is such a p value of any interest? $\endgroup$
    – Michael M
    Sep 18, 2023 at 6:25
  • $\begingroup$ It's just the operating null hypothesis of the business. @MichaelM $\endgroup$ Sep 18, 2023 at 7:07
  • $\begingroup$ So: the answer is: the p value is 0 no matter how you calculate it. $\endgroup$
    – Michael M
    Sep 18, 2023 at 7:12

1 Answer 1

1
$\begingroup$

You say that your null is that the difference in conversion is at least 1 percentage point, but in the comments you also agree that you suspect the change is at least one percentage point. Those two statements are in conflict, so I will do my best to interpret what you need. If I have misunderstood, just let me know and I'll edit where approprioate.

I think you suspect the change is at least one percentage point, so the corresponding null is actually

$$ H0: p_1 - p_0 < 0.01 $$

and the alternative is

$$ HA: p_1 - p_0 \geq 0.01$$

This is a classical one sided test and we can do this with R in the following way given the numbers you provide


Ntrt <- 80000
ytrt <- 70000

Nctrl <- 800100
yctrl <- 70100


p1 <- ytrt/Ntrt
p0 <- yctrl/Nctrl
phat <- (yctrl + ytrt)/(Ntrt + Nctrl)
vrp <- \(x) x*(1-x)

z <- (p1 - p0 - 0.01)/sqrt( vrp(phat)*(1/Ntrt + 1/Nctrl)  )

# This is essentially 0 given your data
pval <- 1 - pnorm(z)

An easier way to do this is to just use prop.test and determine of 0.01 falls within the one sided confidence interval

prop.test(
  c(ytrt, yctrl),
  c(Ntrt, Nctrl),
  correct = F,
  alternative = 'greater'
) -> test

test$conf.int
[1] 0.7853936 1.0000000

Because the confidence interval only contains differences larger than 0.01, you would reject the null in this case.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.