I'm testing the independence of two variables, A and B, stratified by C. A and B are binary variables and C is categorical (5 values). Running Fisher's exact test for A and B (all strata combined), I get:

##          (B)
##      (A) FALSE TRUE
##    FALSE  1841   85
##    TRUE    915   74

OR: 1.75 (1.25 --  2.44), p = 0.0007 *

where OR is the odds ratio (estimate and 95% confidence interval), and * means that p < 0.05.

Running the same test for each stratum (C), I get:

C=1, OR: 2.31 (0.78 --  6.13), p = 0.0815
C=2, OR: 2.75 (1.21 --  6.15), p = 0.0088 *
C=3, OR: 0.94 (0.50 --  1.74), p = 0.8839
C=4, OR: 1.48 (0.77 --  2.89), p = 0.2196
C=5, OR: 3.38 (0.62 -- 34.11), p = 0.1731

Finally, running Cochran-Mantel-Haenszel (CMH) test, using A, B, and C, I get:

OR: 1.56 (1.12 --  2.18), p = 0.0089 *

The result from the CMH test suggests that A and B are not independent at each stratum (p < 0.05); however, most of the within stratum tests were non-significant, which would suggest that we don't have enough evidence to discard that A and B are independent at each stratum.

So, what conclusion is right? How to report the conclusion given those results? Can C be considered a confounding variable or not?

EDIT: I performed the Breslow-Day test for the null hypothesis that the odds ratio is the same across strata, and the p-value was 0.1424.

  • 3
    $\begingroup$ Did you not perform the Cochran-Mantel-Haenszel test precisely because the evidence for an odds ratio different from one might be weak for each stratum considered individually, but strong for all considered together? $\endgroup$ Feb 28, 2014 at 21:32
  • $\begingroup$ I performed CMH because I wanted a single, unified answer, and I wanted to make sure that the effect observed between A and B was not due to C. Am I on the right track? Should I report the statistics for individual strata? $\endgroup$
    – rodrigorgs
    Feb 28, 2014 at 22:24

1 Answer 1


The first test tells you that the odds ratio between A and B, ignoring C, is different from 1. Looking at the stratified analysis helps you decide whether it's all right to ignore C.

The CMH test tells you that the odds ratio between A and B, adjusting for C, is different from one. It returns a weighted average of the stratum-specific odds ratios, so if these are $<1$ in some strata and $>1$ in others, they could cancel out and erroneously tell you there is no association between A and B. So we must test whether it is reasonable to assume that the odds ratios are equal (at the population level) across all the levels of C. The Breslow-Day test of interaction does exactly this, with the null hypothesis that all strata have the same odds ratio, which need not be equal to one. This test is implemented in the EpiR R package. The Breslow-Day p value of .14 means we can make this assumption, so the adjusted odds ratio is legitimate.

But this doesn't help us decide between CMH and Fisher's exact (or Pearson's $\chi^2$) tests. If the Breslow-Day test was significant, you would need to report stratum-specific odds ratios. Since it's not, you need to ask whether it's necessary to adjust for C. Does C "confound" the association between A and B? The heuristic I learned (not a statistical test) was to check whether the proportional difference between the unadjusted and adjusted odds ratios is more than 10%. Here, $\frac{1.75-1.56}{1.75}=0.108$ so CMH is appropriate.

  • $\begingroup$ I edited my question to add the result from the Breslow-Day test (it was 0.14). Therefore, I can say that it is reasonable to assume that the odds ratios are equal? In this case, should I report the Fisher's or the CMH's odds ratio? $\endgroup$
    – rodrigorgs
    Feb 28, 2014 at 23:50
  • 2
    $\begingroup$ Breslow-Day's null hypothesis is "homogeneous odds ratios across strata". Since a p value > 0.05 does not imply the null to be true, you can't assume that the odds ratios are equal. $\endgroup$
    – Michael M
    Mar 1, 2014 at 10:43
  • $\begingroup$ @MichaelMayer: I think you meant to say "the assumption of homogeneous odds ratios isn't discredited, but you shouldn't confuse failing to reject the null with proving the null". $\endgroup$ Mar 1, 2014 at 11:06
  • $\begingroup$ @vafisher: One thing wrong there - the 3rd sentence: Fisher's test still doesn't become appropriate when odds ratios are different across different levels of C. $\endgroup$ Mar 1, 2014 at 11:10
  • $\begingroup$ @Scortchi: good point! $\endgroup$
    – vafisher
    Mar 1, 2014 at 14:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.