What is an appropriate test of $\boldsymbol{H_{0}:RR = 1; H_{\bf{A}}: RR \ne 1}$?

Assume that the data are paired (e.g., from a crossover trial with two treatments, and a dichotomous response), so that:

                 | Treatment Group A      |            |
Treatment Group B|    Positive   Negative |      Total |
        Positive |         a           b  |        a+b |
        Negative |         c           d  |        c+d |
           Total |       a+c         b+d  |          n |

Assuming each pair contains one person from Treatment Group A, and one person from Treatment Group B:

Kind of pair    Count
A Pos & B Pos      a
A Neg & B Pos      b
A Pos & B Neg      c
A Neg & B Neg      d

So relative risk of a positive outcome for Treatment Group B vs. Treatment Group A (RR) is given by:

$$RR = \frac{\frac{a+b}{n}}{\frac{a+c}{n}} = \frac{a+b}{a+c}$$

There is a reasonably traceable literature on tests for equivalence for this kind of relative risk (e.g., $H_{01}: RR \ge \delta$ or $H_{02}: RR\le \delta^{-1}$), see, for example, Tang Tan and Chan (2003). However, I am finding the literature for tests for difference as I have outlined in my question to be a tad elusive.

McNemar's test, which is also for paired dichotomous outcomes, is a test of odds ratios, and only examines discordant pairs, so I suspect there should be a separate RR test… am I in error? My biostatistics and epidemiology textbooks are mum.

Edit: In response to a comment by @gung, I would like to clarify that I am not (yet) persuaded that an odds ratio test is appropriate for a relative risk test. Consider:

  if $a = 10; b= 6; c = 12; d = 72; n = 100$ then $RR=.723$, but $OR=0.5$, while

  if $a = 44; b= 6; c = 12; d = 38; n = 100$ then $RR=.892$, but still $OR=0.5$, and

  if $a = 1000; b= 6; c = 12; d = 72; n = 1090$ then $RR=.994$, but yet again, $OR=0.5$.

So a isn't contributing independent information that differentiates RR from OR? And shouldn't RR have it's own form of test statistic?

McNemar, Q. (1947). Note on the sampling error of the difference between correlated proportions or percentages. Psychometrika, 12:153–157

Tang, N.-S., Tang, M.-L., and Chan, I. S. F. (2003). On tests of equivalence via non-unity relative risk for matched-pair design. Statistics In Medicine, 22:1217–1233.

  • 3
    $\begingroup$ I don't see how a can contribute meaningful, independent information here. If McNemar's test / the OR is significant, it seems to me that the RR must be significant as well. So you could just use McN, & then form & interpret the RRs. I can write this up as an official answer, if you like, or I can just leave it as this comment. $\endgroup$ – gung - Reinstate Monica Apr 22 '17 at 1:11
  • 1
    $\begingroup$ @gung $RR \ne OR$. (Notably, in these data $OR = \frac{b}{c}$) Which is not to say your logic does not hold. So, I wouldn't mind a more detailed write up explaining why for an answer. :) $\endgroup$ – Alexis Apr 22 '17 at 3:14
  • 2
    $\begingroup$ There is a rare event assumption. If the outcome is not common, the OR approximates the RR. Also the statistical significance of the OR implies statistical significance of the RR. That is: if the OR $\ne$ 1, then the RR $\ne$ 1. $\endgroup$ – AdamO Jun 12 '17 at 21:50
  • 3
    $\begingroup$ @Alexis there may be some confusion about the objective of qualitative inference. You state $RR \approx 1$ but it is certain $RR < 1$. It is easy to prove that if $p_1 < p_2$ then $p_1/(1-p_1) < p_2/(1-p_2)$. So you can infer the direction of the relationship but not the magnitude of the effect: a serious limitation admittedly. $\endgroup$ – AdamO Jun 12 '17 at 23:32
  • 1
    $\begingroup$ @Alexis: It's one of the advantages of the much-maligned nil-null hypothesis test that it doesn't require a parametric specification of the way in which two groups might be unalike. So a hypothesized relative risk of one, an odds ratio of one, a difference in rates of zero, &c. - all come to the same thing. $\endgroup$ – Scortchi - Reinstate Monica Jun 26 '17 at 10:18

Gung's suggestion is correct: if the objective is merely stating and testing a hypothesis, the McNemar's test is correct. Like the intuition from modeling independent data, even in dependent data: when the OR != 1, the RR != 1 and the discordant pair probability != 0.5. A limitation of this type of analysis is that you do not explicitly come up with a confidence interval for the relative risk. I might recommend two approaches to get around this:

  1. Use a mixed model
  2. Use a mid-p probability model and transform to a relative risk.

The median-unbiased mid-p ratio estimators, their CIs, and p-values are invariant to transformations that preserve order (1-1). So like in McNemar's test, we can focus on modeling an event probability among discordant pairs. When the null hypothesis is true, $p=0.5$. But this conditional probability can be transformed to a relative risk by rescaling it according to the number of discordant pairs.

Here's a simulation using a simply random intercepts-type data generating process with a mid-p estimator. We see the coverage of 95% CIs for the relative risk is approximately correct.

do.one <- function(n, sig0, rr) {
  risk <- exp(rep(sig0, each=2))*rep(c(1,rr),n)
  y <- rbinom(n*2, 1, pmin(risk, 1))
  y <- matrix(y, ncol=2, byrow=T)
  y <- table(factor(y[, 1], levels=0:1), factor(y[, 2], levels=0:1))
  dp <- c(y[2,1], y[1,2])
  ndp <- rep(sum(dp), 2)
  out <- rateratio.midp(x = dp, y=rep(sum(dp), 2))
  ci <- out$measure[2, 2:3]
  ci[1] < rr & ci[2] > rr

test <- replicate(10000, do.one(5000, sig0=rnorm(5000, -4), rr=1.2))
mean(test, na.rm=T) ## NA cases where no events

Gives approximately the correct 95% coverage

>     mean(test, na.rm=T) ## NA cases where no events
[1] 0.9457

Fagerland, M.W.; Lydersen, S.; Laake, P. (2013). "The McNemar test for binary matched-pairs data: mid-p and asymptotic are better than exact conditional". BMC Medical Research Methodology. 13: 91

Kenneth J. Rothman, Sander Greenland, and Timothy Lash (2008), Modern Epidemiology, Lippincott-Raven Publishers

Kenneth J. Rothman (2012), Epidemiology: An Introduction, Oxford University Press


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.