# Confidence interval around the ratio of two proportions

I have two proportions (e.g., clickthrough rate (CTR) on a link in a control layout, and CTR on a link in an experimental layout), and I want to calculate a 95% confidence interval around the ratio of these proportions.

How do I do this? I know I can use the delta method to calculate the variance of this ratio, but I'm not sure what to do besides that. What should I use as the midpoint of the confidence interval (my observed ratio, or the expected ratio which is different), and how many standard deviations around this ratio should I take?

Should I be using the delta method variance at all? (I don't really care about the variance, just a confidence interval.) Should I use Fieller's Theorem, using Case 1 (since I'm doing proportions, I guess I satisfy the normal distribution requirement)? Should I just calculate a bootstrap sample?

• You have a fundamental problem: most proportions have a positive chance of being zero, whence the ratio (of independent proportions) has a positive chance of being undefined. This can present severe difficulties for approximate methods (like the delta method) and suggests that normal approximations should be viewed more sceptically and tested more rigorously than usual. – whuber Jan 18 '12 at 21:16
• Joseph L. Fleiss, Bruce Levin, Myunghee Cho Paik: Statistical Methods for Rates and Proportions[1] discusses the Relative Risk, which is a quotient of two rates. I don't have the book, so I can only go by the subject index and table of contents, but maybe your library has it. [1]: onlinelibrary.wiley.com/book/10.1002/0471445428 – cbeleites unhappy with SX Jan 19 '12 at 12:26
• Surely a percentile bootstrap would be the best method? – Peter Ellis Jan 19 '12 at 18:44

If there are $x_1$ and $x_2$ successes in the two groups out of totals $n_1$ and $n_2$, then the obvious estimate for the ratio of proportions is $$\hat\theta = \frac{x_1/n_1}{x_2/n_2}.$$
Using the delta method and assuming the two groups are independent and the successes are binomially distributed, you can show that $$\operatorname{Var}(\log \hat\theta) = 1/x_1 - 1/n_1 +1/x_2 - 1/n_2.$$ Taking the square-root of this gives the standard error $\operatorname{SE}(\log \hat\theta)$. Assuming that $\log \hat\theta$ is normally distributed, a 95% confidence interval for $\log \theta$ is $$\log \hat\theta \pm 1.96 \operatorname{SE}(\log \hat\theta).$$ Exponentiating this gives a 95% confidence interval for the ratio of proportions $\theta$ as $$\hat\theta \exp\left[ \pm1.96 \operatorname{SE}(\log\hat\theta)\right].$$
• This works great provided $n_1$ and $n_2$ are large (several hundred or more) and $n_1 p_1$ and $n_2 p_2$ are not too small (c. $10$ or more). Otherwise, the interval tends to be too large. It also needs some way to treat the cases $x_2=0$ and $x_i=n_i$. It turns out both issues can be addressed with a continuity-correction-like approach: add $1/2$ to both the $x_i$, add $1$ to both the $n_i$, and proceed. Then this CI is surprisingly good provided both of the $p_i n_i$ are $4$ or greater, regardless of the sizes of the $n_i$. – whuber Jan 18 '12 at 23:25
• Interesting question, raegtin. In this case, no: I experimented to find a suitable starting value (that's the meaning of "it turns out that"). 1/2 is not universally valid; for certain combinations of $x_i$ and $n_i$, other values will work slightly better. A theoretical study of the distribution of the estimator might suggest a different starting value. – whuber Jan 20 '12 at 19:37