2
$\begingroup$

I am looking for a significance test to compare between two association matrices (contingency tables) - preferably to test whether the odds ratio is the same. I have looked at some questions here but they don't provide the answer I am looking for.

For example, I would like to compare the following tables:

Data

(Apologies the tables are presented as an image. I couldn't get MathJax to render them as tables)

What would be an appropriate test to use?

$\endgroup$
2
  • $\begingroup$ What is your hypothesis? That the odds ratio is the same in both tables, or ...? $\endgroup$ Jul 25, 2023 at 1:13
  • $\begingroup$ @kjetilbhalvorsen - yes, sorry, I forgot to add that in my haste - testing that the odds ratio is the same in both tables. I will edit my question. $\endgroup$
    – anna6931
    Jul 25, 2023 at 1:56

2 Answers 2

1
$\begingroup$

Firstly, notating the contingency table as:

$$\begin{array}{c|c|c|} & \text{True} & \text{False} \\ \hline \text{Before} & a & b\\ \hline \text{After} & c & d \\ \hline \end{array}$$

the Odds Ratio $(OR)$ is:

$$OR=\frac{ad}{bc}$$

Taking your data, for Test 1, $OR_1=7.813$ and for Test 2, $OR_2=0.624$.

The standard error $(SE)$ for the Odds Ratio can be approximated by:

$$SE=\sqrt{\frac{1}{a}+\frac{1}{b}+\frac{1}{c}+\frac{1}{d}}$$

For Test 1, $SE_1=0.515$ and for Test 2 $SE_2=0.480$

The confidence interval of the $OR$ is taken as $e^{\ln(⁡OR)\, \pm\, z\, SE}$.

If we set $z=1.96$ (i.e. 95% confidence level), we have the following confidence intervals:

$$\begin{array}{c|c|c|} & \text{Lower} & \text{Upper} \\ \hline \text{Test 1 CI} & 2.848 & 21.428\\ \hline \text{Test 2 CI} & 0.243 & 1.599 \\ \hline \end{array}$$

As the confidence interval for the tests do not overlap, we can conclude there is a significant difference in the odds ratio between the tests at the 95% confidence level.

One question remains: what is the p-value? (assuming that is what you are after). To find this, we need to determine the point where the confidence interval from one test intersects the confidence interval from the other test. This occurs when:

$$e^{\ln(⁡OR_i)\, -\, z\, SE_i}=e^{\ln(⁡OR_j)\, +\, z\, SE_j}$$ For convention, let $i$ be the test with the greatest $OR$ and $j$ be the test with the lowest $OR$. This can be solved for the absolute value of $z$ as follows:

$$z=\left| \frac{\ln OR_i-\ln OR_j}{SE_i+SE_j} \right |$$

Inserting the relevant data $$z=\left | \frac{\ln(7.813)-\ln⁡(0.624)}{0.515+0.480} \right |\,=\,2.541$$ Taking the standard normal distribution, this equates to a p-value (2-sided) of 0.011.

$\endgroup$
1
  • $\begingroup$ Thanks for your answer. I can follow the approach you have taken. $\endgroup$
    – anna6931
    Jul 29, 2023 at 13:18
3
$\begingroup$

You can do this via logistic regression. First we must put your data in a convenient form, in R:

yourdata <- "Test  Period Outcome w
             test1 before TRUE    25
             test1 before FALSE   14
             test1 after  TRUE    8
             test1 after  FALSE   35
             test2 before TRUE    13
             test2 before FALSE   18
             test2 after  TRUE    22
             test2 after  FALSE   19 "

dat <- read.table(textConnection(yourdata), header=TRUE, 
                  stringsAsFactors=TRUE)

Then building a logistic regression model, with an interaction term that represents the difference in odds ratio between the two tables:

mod0 <- glm(Outcome ~ Test * Period, weight=w, data=dat, family=binomial)
> summary(mod0)

Call:
glm(formula = Outcome ~ Test * Period, family = binomial, data = dat, 
    weights = w)

Deviance Residuals: 
     1       2       3       4       5       6       7       8  
 4.715  -5.356   5.187  -3.796   4.753  -4.424   5.234  -5.406  

Coefficients:
                       Estimate Std. Error z value Pr(>|z|)    
(Intercept)             -1.4759     0.3919  -3.766 0.000166 ***
Testtest2                1.6225     0.5017   3.234 0.001219 ** 
Periodbefore             2.0557     0.5148   3.993 6.51e-05 ***
Testtest2:Periodbefore  -2.5278     0.7040  -3.591 0.000330 ***
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

(Dispersion parameter for binomial family taken to be 1)

    Null deviance: 211.38  on 7  degrees of freedom
Residual deviance: 191.02  on 4  degrees of freedom
AIC: 199.02

Number of Fisher Scoring iterations: 5

EDIT There might be other approaches. You have two contingency tables, if you lay one over the other (using Test as the layer variable) you get a three-way contingency table, that could be a search term. There are specific hypothesis tests for that situation, but probably an approach like above using generalized linear models is preferred today. Some links you could look at:

$\endgroup$
3
  • 2
    $\begingroup$ +1 This is what I did for contingency table analysis a few years ago, and I am glad to see someone reputable saying to do the same. I also did a chunk test of the test2 variable and the interaction as a full test of contingency table equivalence. I didn’t care what the odds ratios in the tables were, as long as they differed (more or less). $\endgroup$
    – Dave
    Jul 26, 2023 at 22:34
  • 2
    $\begingroup$ Appreciate taking the time to answer. I am still learning advanced stats and R so some of this is new for me. $\endgroup$
    – anna6931
    Jul 27, 2023 at 21:27
  • 1
    $\begingroup$ I added some info ... $\endgroup$ Jul 27, 2023 at 22:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.