22
$\begingroup$

I have two proportions (e.g., clickthrough rate (CTR) on a link in a control layout, and CTR on a link in an experimental layout), and I want to calculate a 95% confidence interval around the ratio of these proportions.

How do I do this? I know I can use the delta method to calculate the variance of this ratio, but I'm not sure what to do besides that. What should I use as the midpoint of the confidence interval (my observed ratio, or the expected ratio which is different), and how many standard deviations around this ratio should I take?

Should I be using the delta method variance at all? (I don't really care about the variance, just a confidence interval.) Should I use Fieller's Theorem, using Case 1 (since I'm doing proportions, I guess I satisfy the normal distribution requirement)? Should I just calculate a bootstrap sample?

Improve this question
$\endgroup$
3
  • 1
    $\begingroup$ You have a fundamental problem: most proportions have a positive chance of being zero, whence the ratio (of independent proportions) has a positive chance of being undefined. This can present severe difficulties for approximate methods (like the delta method) and suggests that normal approximations should be viewed more sceptically and tested more rigorously than usual. $\endgroup$
    – whuber ♦
    Commented Jan 18, 2012 at 21:16
  • 1
    $\begingroup$ Joseph L. Fleiss, Bruce Levin, Myunghee Cho Paik: Statistical Methods for Rates and Proportions[1] discusses the Relative Risk, which is a quotient of two rates. I don't have the book, so I can only go by the subject index and table of contents, but maybe your library has it. [1]: onlinelibrary.wiley.com/book/10.1002/0471445428 $\endgroup$
    – cbeleites
    Commented Jan 19, 2012 at 12:26
  • $\begingroup$ Surely a percentile bootstrap would be the best method? $\endgroup$
    – Peter Ellis
    Commented Jan 19, 2012 at 18:44

1 Answer 1

Reset to default
25
$\begingroup$

The standard way to do this in epidemiology (where a ratio of proportions is usually referred to as a risk ratio) is to first log-transform the ratio, calculate a confidence interval on the log scale using the delta method and assuming a normal distribution, then transform back. This works better in moderate sample sizes than using the delta method on the untransformed scale, though it will still behave poorly if the number of events in either group is very small, and fails completely if there are no events in either group.

If there are $x_1$ and $x_2$ successes in the two groups out of totals $n_1$ and $n_2$, then the obvious estimate for the ratio of proportions is $$\hat\theta = \frac{x_1/n_1}{x_2/n_2}.$$

Using the delta method and assuming the two groups are independent and the successes are binomially distributed, you can show that $$\operatorname{Var}(\log \hat\theta) = 1/x_1 - 1/n_1 +1/x_2 - 1/n_2.$$ Taking the square-root of this gives the standard error $\operatorname{SE}(\log \hat\theta)$. Assuming that $\log \hat\theta$ is normally distributed, a 95% confidence interval for $\log \theta$ is $$\log \hat\theta \pm 1.96 \operatorname{SE}(\log \hat\theta).$$ Exponentiating this gives a 95% confidence interval for the ratio of proportions $\theta$ as $$\hat\theta \exp\left[ \pm1.96 \operatorname{SE}(\log\hat\theta)\right].$$

Improve this answer
$\endgroup$
5
  • 7
    $\begingroup$ This works great provided $n_1$ and $n_2$ are large (several hundred or more) and $n_1 p_1$ and $n_2 p_2$ are not too small (c. $10$ or more). Otherwise, the interval tends to be too large. It also needs some way to treat the cases $x_2=0$ and $x_i=n_i$. It turns out both issues can be addressed with a continuity-correction-like approach: add $1/2$ to both the $x_i$, add $1$ to both the $n_i$, and proceed. Then this CI is surprisingly good provided both of the $p_i n_i$ are $4$ or greater, regardless of the sizes of the $n_i$. $\endgroup$
    – whuber ♦
    Commented Jan 18, 2012 at 23:25
  • $\begingroup$ @whuber: "continuity-correction-like approach" -- is the use of 1/2 in particular a common trick? (As opposed to some other small pseudocount.) The way you phrased it makes 1/2 sound principled in some way =) -- is it? $\endgroup$
    – raegtin
    Commented Jan 20, 2012 at 19:12
  • $\begingroup$ Interesting question, raegtin. In this case, no: I experimented to find a suitable starting value (that's the meaning of "it turns out that"). 1/2 is not universally valid; for certain combinations of $x_i$ and $n_i$, other values will work slightly better. A theoretical study of the distribution of the estimator might suggest a different starting value. $\endgroup$
    – whuber ♦
    Commented Jan 20, 2012 at 19:37
  • $\begingroup$ Why is square-root of variance standard error in this case, not standard deviation? $\endgroup$
    – Mikko
    Commented Mar 6, 2013 at 10:04
  • 4
    $\begingroup$ @onestop Is this implemented in any R package? $\endgroup$
    – Bogdan Vasilescu
    Commented Jul 23, 2013 at 19:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.