1
$\begingroup$

I am interested in running A/B tests on differences in proportions and reporting the confidence intervals to convey the precision of the test. Tests would be of the form $H_{o}: p(a) \geq p(b)$ vs $H_{a}: p(a) < p(b)$. I am using the test statistic:

$$ \begin{equation} \frac{p(b) - p(a)}{\sqrt{(p*(1-p)*\left(\frac{1}{n_{a}} + \frac{1}{n_{b}}\right)}} \end{equation} $$

where

$$ \begin{equation} p=\frac{n_{b}*p(b) + n_{a}*p(a)}{n_{a}+ n_{b}} \end{equation} $$

Here is some example data to illustrate the problem I'm having: $n_{a}=n_{b}=20561$, $p(a)=\frac{3151}{20561}$, $p(b)=\frac{3280}{20561}$. If I am testing at $\alpha=0.05$ I would reject the null hypothesis because the test statistic is greater than 1.645

However, I don't understand how to report the confidence interval. This is a one-sided hypothesis test, and one sided tests usually have confidence intervals of the form $(\overline{X} - z_{alpha}*se, \infty)$. Since the difference in proportions is at most 1, the one sided confidence interval could be reduced to $(\overline{X} - z_{alpha}*se, 1)$. However, this gives little indication of the precision of the test.

If I calculate a 2-sided confidence interval at level $\alpha$, then I get the strange result of rejecting the null hypothesis, but having the confidence interval (-0.0007469738, 0.013295) which includes zero.

The solution I like the best is calculating the confidence level for $2\alpha$. Then I never get a negative lower bound in my confidence interval if I reject the null hypothesis, but it seems strange to perform the test at level $\alpha$ and then to construct the confidence interval at $2\alpha$

I haven't found much on this topic. It is similar to this Hypothesis tests and confidence intervals. Greg Snow's answer states that some statistical software packages would compute a $90\%$ confidence interval for $\alpha=0.05$. This gives the result that I am most satisfied with, but I am wondering how I should report the results and if this is justified.

For my example, could I say that I reject the null hypothesis with $95\%$ confidence and that p(b) is greater than p(a) by 0.006274014 with a $90\%$ confidence interval of (0.0003818153, 0.01216621)?

$\endgroup$
  • $\begingroup$ "could I say that I reject the null hypothesis with 95% confidence" -- well you could say it, but it would be wrong. $\endgroup$ – Glen_b Jan 31 '16 at 0:39
  • $\begingroup$ ok, it would be better to say I reject the null at the 5% significance level and that my estimate is that p(b) is greater than p(a) by 0.006274014 $\endgroup$ – lever Jan 31 '16 at 1:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.