I have two proportions (e.g., clickthrough rate (CTR) on a link in a control layout, and CTR on a link in an experimental layout), and I want to calculate a 95% confidence interval around the ratio of these proportions.

How do I do this? I know I can use the delta method to calculate the variance of this ratio, but I'm not sure what to do besides that. What should I use as the midpoint of the confidence interval (my observed ratio, or the expected ratio which is different), and how many standard deviations around this ratio should I take?

Should I be using the delta method variance at all? (I don't really care about the variance, just a confidence interval.) Should I use Fieller's Theorem, using Case 1 (since I'm doing proportions, I guess I satisfy the normal distribution requirement)? Should I just calculate a bootstrap sample?

  • 1
    $\begingroup$ You have a fundamental problem: most proportions have a positive chance of being zero, whence the ratio (of independent proportions) has a positive chance of being undefined. This can present severe difficulties for approximate methods (like the delta method) and suggests that normal approximations should be viewed more sceptically and tested more rigorously than usual. $\endgroup$ – whuber Jan 18 '12 at 21:16
  • 1
    $\begingroup$ Joseph L. Fleiss, Bruce Levin, Myunghee Cho Paik: Statistical Methods for Rates and Proportions[1] discusses the Relative Risk, which is a quotient of two rates. I don't have the book, so I can only go by the subject index and table of contents, but maybe your library has it. [1]: onlinelibrary.wiley.com/book/10.1002/0471445428 $\endgroup$ – cbeleites unhappy with SX Jan 19 '12 at 12:26
  • $\begingroup$ Surely a percentile bootstrap would be the best method? $\endgroup$ – Peter Ellis Jan 19 '12 at 18:44

The standard way to do this in epidemiology (where a ratio of proportions is usually referred to as a risk ratio) is to first log-transform the ratio, calculate a confidence interval on the log scale using the delta method and assuming a normal distribution, then transform back. This works better in moderate sample sizes than using the delta method on the untransformed scale, though it will still behave poorly if the number of events in either group is very small, and fails completely if there are no events in either group.

If there are $x_1$ and $x_2$ successes in the two groups out of totals $n_1$ and $n_2$, then the obvious estimate for the ratio of proportions is $$\hat\theta = \frac{x_1/n_1}{x_2/n_2}.$$

Using the delta method and assuming the two groups are independent and the successes are binomially distributed, you can show that $$\operatorname{Var}(\log \hat\theta) = 1/x_1 - 1/n_1 +1/x_2 - 1/n_2.$$ Taking the square-root of this gives the standard error $\operatorname{SE}(\log \hat\theta)$. Assuming that $\log \hat\theta$ is normally distributed, a 95% confidence interval for $\log \theta$ is $$\log \hat\theta \pm 1.96 \operatorname{SE}(\log \hat\theta).$$ Exponentiating this gives a 95% confidence interval for the ratio of proportions $\theta$ as $$\hat\theta \exp\left[ \pm1.96 \operatorname{SE}(\log\hat\theta)\right].$$

  • 7
    $\begingroup$ This works great provided $n_1$ and $n_2$ are large (several hundred or more) and $n_1 p_1$ and $n_2 p_2$ are not too small (c. $10$ or more). Otherwise, the interval tends to be too large. It also needs some way to treat the cases $x_2=0$ and $x_i=n_i$. It turns out both issues can be addressed with a continuity-correction-like approach: add $1/2$ to both the $x_i$, add $1$ to both the $n_i$, and proceed. Then this CI is surprisingly good provided both of the $p_i n_i$ are $4$ or greater, regardless of the sizes of the $n_i$. $\endgroup$ – whuber Jan 18 '12 at 23:25
  • $\begingroup$ @whuber: "continuity-correction-like approach" -- is the use of 1/2 in particular a common trick? (As opposed to some other small pseudocount.) The way you phrased it makes 1/2 sound principled in some way =) -- is it? $\endgroup$ – raegtin Jan 20 '12 at 19:12
  • $\begingroup$ Interesting question, raegtin. In this case, no: I experimented to find a suitable starting value (that's the meaning of "it turns out that"). 1/2 is not universally valid; for certain combinations of $x_i$ and $n_i$, other values will work slightly better. A theoretical study of the distribution of the estimator might suggest a different starting value. $\endgroup$ – whuber Jan 20 '12 at 19:37
  • $\begingroup$ Why is square-root of variance standard error in this case, not standard deviation? $\endgroup$ – Mikko Mar 6 '13 at 10:04
  • 3
    $\begingroup$ @onestop Is this implemented in any R package? $\endgroup$ – Bogdan Vasilescu Jul 23 '13 at 19:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.