# How to interpret Cochran-Mantel-Haenszel test?

I'm testing the independence of two variables, A and B, stratified by C. A and B are binary variables and C is categorical (5 values). Running Fisher's exact test for A and B (all strata combined), I get:

##          (B)
##      (A) FALSE TRUE
##    FALSE  1841   85
##    TRUE    915   74

OR: 1.75 (1.25 --  2.44), p = 0.0007 *


where OR is the odds ratio (estimate and 95% confidence interval), and * means that p < 0.05.

Running the same test for each stratum (C), I get:

C=1, OR: 2.31 (0.78 --  6.13), p = 0.0815
C=2, OR: 2.75 (1.21 --  6.15), p = 0.0088 *
C=3, OR: 0.94 (0.50 --  1.74), p = 0.8839
C=4, OR: 1.48 (0.77 --  2.89), p = 0.2196
C=5, OR: 3.38 (0.62 -- 34.11), p = 0.1731


Finally, running Cochran-Mantel-Haenszel (CMH) test, using A, B, and C, I get:

OR: 1.56 (1.12 --  2.18), p = 0.0089 *


The result from the CMH test suggests that A and B are not independent at each stratum (p < 0.05); however, most of the within stratum tests were non-significant, which would suggest that we don't have enough evidence to discard that A and B are independent at each stratum.

So, what conclusion is right? How to report the conclusion given those results? Can C be considered a confounding variable or not?

EDIT: I performed the Breslow-Day test for the null hypothesis that the odds ratio is the same across strata, and the p-value was 0.1424.

• Did you not perform the Cochran-Mantel-Haenszel test precisely because the evidence for an odds ratio different from one might be weak for each stratum considered individually, but strong for all considered together? Feb 28, 2014 at 21:32
• I performed CMH because I wanted a single, unified answer, and I wanted to make sure that the effect observed between A and B was not due to C. Am I on the right track? Should I report the statistics for individual strata? Feb 28, 2014 at 22:24

The CMH test tells you that the odds ratio between A and B, adjusting for C, is different from one. It returns a weighted average of the stratum-specific odds ratios, so if these are $<1$ in some strata and $>1$ in others, they could cancel out and erroneously tell you there is no association between A and B. So we must test whether it is reasonable to assume that the odds ratios are equal (at the population level) across all the levels of C. The Breslow-Day test of interaction does exactly this, with the null hypothesis that all strata have the same odds ratio, which need not be equal to one. This test is implemented in the EpiR R package. The Breslow-Day p value of .14 means we can make this assumption, so the adjusted odds ratio is legitimate.
But this doesn't help us decide between CMH and Fisher's exact (or Pearson's $\chi^2$) tests. If the Breslow-Day test was significant, you would need to report stratum-specific odds ratios. Since it's not, you need to ask whether it's necessary to adjust for C. Does C "confound" the association between A and B? The heuristic I learned (not a statistical test) was to check whether the proportional difference between the unadjusted and adjusted odds ratios is more than 10%. Here, $\frac{1.75-1.56}{1.75}=0.108$ so CMH is appropriate.