Comparing two association matrices (contingency tables)

Question

I am looking for a significance test to compare between two association matrices (contingency tables) - preferably to test whether the odds ratio is the same. I have looked at some questions here but they don't provide the answer I am looking for.

For example, I would like to compare the following tables:

(Apologies the tables are presented as an image. I couldn't get MathJax to render them as tables)

What would be an appropriate test to use?

Answer 1

Firstly, notating the contingency table as:

$$\begin{array}{c|c|c|} & \text{True} & \text{False} \\ \hline \text{Before} & a & b\\ \hline \text{After} & c & d \\ \hline \end{array}$$

the Odds Ratio $(OR)$ is:

$$OR=\frac{ad}{bc}$$

Taking your data, for Test 1, $OR_1=7.813$ and for Test 2, $OR_2=0.624$.

The standard error $(SE)$ for the Odds Ratio can be approximated by:

$$SE=\sqrt{\frac{1}{a}+\frac{1}{b}+\frac{1}{c}+\frac{1}{d}}$$

For Test 1, $SE_1=0.515$ and for Test 2 $SE_2=0.480$

The confidence interval of the $OR$ is taken as $e^{\ln(⁡OR)\, \pm\, z\, SE}$.

If we set $z=1.96$ (i.e. 95% confidence level), we have the following confidence intervals:

$$\begin{array}{c|c|c|} & \text{Lower} & \text{Upper} \\ \hline \text{Test 1 CI} & 2.848 & 21.428\\ \hline \text{Test 2 CI} & 0.243 & 1.599 \\ \hline \end{array}$$

As the confidence interval for the tests do not overlap, we can conclude there is a significant difference in the odds ratio between the tests at the 95% confidence level.

One question remains: what is the p-value? (assuming that is what you are after). To find this, we need to determine the point where the confidence interval from one test intersects the confidence interval from the other test. This occurs when:

$$e^{\ln(⁡OR_i)\, -\, z\, SE_i}=e^{\ln(⁡OR_j)\, +\, z\, SE_j}$$ For convention, let $i$ be the test with the greatest $OR$ and $j$ be the test with the lowest $OR$. This can be solved for the absolute value of $z$ as follows:

$$z=\left| \frac{\ln OR_i-\ln OR_j}{SE_i+SE_j} \right |$$

Inserting the relevant data $$z=\left | \frac{\ln(7.813)-\ln⁡(0.624)}{0.515+0.480} \right |\,=\,2.541$$ Taking the standard normal distribution, this equates to a p-value (2-sided) of 0.011.

Answer 2

You can do this via logistic regression. First we must put your data in a convenient form, in R:

yourdata <- "Test  Period Outcome w
             test1 before TRUE    25
             test1 before FALSE   14
             test1 after  TRUE    8
             test1 after  FALSE   35
             test2 before TRUE    13
             test2 before FALSE   18
             test2 after  TRUE    22
             test2 after  FALSE   19 "

dat <- read.table(textConnection(yourdata), header=TRUE, 
                  stringsAsFactors=TRUE)

Then building a logistic regression model, with an interaction term that represents the difference in odds ratio between the two tables:

mod0 <- glm(Outcome ~ Test * Period, weight=w, data=dat, family=binomial)
> summary(mod0)

Call:
glm(formula = Outcome ~ Test * Period, family = binomial, data = dat, 
    weights = w)

Deviance Residuals: 
     1       2       3       4       5       6       7       8  
 4.715  -5.356   5.187  -3.796   4.753  -4.424   5.234  -5.406  

Coefficients:
                       Estimate Std. Error z value Pr(>|z|)    
(Intercept)             -1.4759     0.3919  -3.766 0.000166 ***
Testtest2                1.6225     0.5017   3.234 0.001219 ** 
Periodbefore             2.0557     0.5148   3.993 6.51e-05 ***
Testtest2:Periodbefore  -2.5278     0.7040  -3.591 0.000330 ***
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

(Dispersion parameter for binomial family taken to be 1)

    Null deviance: 211.38  on 7  degrees of freedom
Residual deviance: 191.02  on 4  degrees of freedom
AIC: 199.02

Number of Fisher Scoring iterations: 5

EDIT There might be other approaches. You have two contingency tables, if you lay one over the other (using Test as the layer variable) you get a three-way contingency table, that could be a search term. There are specific hypothesis tests for that situation, but probably an approach like above using generalized linear models is preferred today. Some links you could look at:

• https://cran.r-project.org/web/packages/samplesizeCMH/vignettes/samplesizeCMH-introduction.html

• https://web.pdx.edu/~newsomj/cdaclass/ho_three-way.pdf