9
$\begingroup$

In a comment here, @gung wrote,

I believe they can overlap a little (maybe ~25%) & still be significant at the 5% level. Remember that the 95% CI you see is for the individual OR, but the test of 2 ORs is about the difference between them. However, if they don't overlap at all, then they are definitely significantly different, & if the 95% CI's overlap the other OR point estimate, they definitely don't.

Does anyone have citations for the above statement? A reviewer wants me to calculate if two odds ratios are significantly different to each other.

$\endgroup$
  • $\begingroup$ Why not just calculate the significance of the difference between two odds ratios directly? Why would you wan to measure the overlap of the 95% CIs & try to get the significance from that? $\endgroup$ – gung - Reinstate Monica Mar 25 '15 at 1:18
  • $\begingroup$ What is the equation to do this? $\endgroup$ – cpjh10 Mar 25 '15 at 15:30
  • $\begingroup$ To test for the difference of two odds ratios? Do you know the odds ratios & the Ns they are based on? Do you have access to the original data? $\endgroup$ – gung - Reinstate Monica Mar 25 '15 at 15:38
  • $\begingroup$ Yes, it was a multilevel logistic regression (the bernoulli option using the HLM software). SO I have the ORs and the Ns from that analysis. $\endgroup$ – cpjh10 Mar 25 '15 at 15:46
  • $\begingroup$ The output from the analysis should tell you if they are significantly different, or you should be able to get your software to give that to you by adding some option. Do you have the SEs for the ORs? Are they independent, or do you have an estimate of the covariance of their sampling distributions? $\endgroup$ – gung - Reinstate Monica Mar 25 '15 at 15:54
3
$\begingroup$

From your two logistic regression models, you should have parameter estimates, $\hat\beta_{11}$ and $\hat\beta_{12}$ (where the second subscript refers to the model), and their standard errors. Note that these are on the scale of the log odds and that this is better—there is no need to convert them to odds ratios. If your $N$s are sufficient, these will be normally distributed, as @ssdecontrol explained. The Wald tests that come standard with logistic regression output assume they are normally distributed, for example. In addition, since they came from different models with different data, we can treat them as independent. If you want to test if they are equal, this is simply testing a linear combination of normally distributed parameter estimates, which is a pretty standard thing to do. You can calculate a test statistic as follows:
$$ Z = \frac{\hat\beta_{12}-\hat\beta_{11}}{\sqrt{SE(\hat\beta_{12})^2 + SE(\hat\beta_{11})^2}} $$ The resulting $Z$ statistic can be compared to a standard normal distribution to compute the $p$-value.

The quote about confidence intervals is somewhat heuristic in nature (even though correct). You should not try to use that to calculate significance.

$\endgroup$
9
$\begingroup$

Odds ratios are asymptotically Gaussian.

Therefore their difference, as long as they are independent, is also asymptotically Gaussian, because the linear combination of independent Gaussian r.v.s is itself Gaussian.

These are both fairly well-known and shouldn't require a citation. But just for assurance, both of those links are based on "authoritative" sources.

$\endgroup$
  • 8
    $\begingroup$ Log(odds ratios) tends to be closer to Gaussian in finite samples: an odds ratio cannot be smaller than 0, but a log(odds ratio) can. $\endgroup$ – Maarten Buis Mar 25 '15 at 8:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.