In a recent paper Norton et al. (2018)$^{[2]}$ state that
Different odds ratios from the same study cannot be compared when the statistical models that result in odds ratio estimates have different explanatory variables because each model has a different arbitrary scaling factor. Nor can the magnitude of the odds ratio from one study be compared with the magnitude of the odds ratio from another study, because different samples and different model specifications will have different arbitrary scaling factors. A further implication is that the magnitudes of odds ratios of a given association in multiple studies cannot be synthesized in a meta-analysis.
A small simulation illustrates this (R code is at the bottom of the question). Suppose the true model is: $$ \mathrm{logit}(y_{i})=1 + \log(2)x_{1i} + \log(2.5)x_{2i} + \log(3)x_{3i} + 0x_{4i} $$ Supposed we're interested in the effect for $x_{1}$ (i.e. $x_{1}$ is the focal predictor). Imagine further that the same data generated by the above model is analyzed by four different researchers using a logistic regression. Researcher 1 only includes $x_{1}$ as a covariate, researcher 2 includes both $x_{1}$ and $x_{2}$ and so forth. The average simulated estimates of the odds ratio for $x_{1}$ of the four researchers were:
res_1 res_2 res_3 res_4
1.679768 1.776200 2.002157 2.004077
It's apparent that only researchers 3 and 4 get the correct odds ratio of about $2$ whereas researchers 1 and 2 do not. This does not happen in linear regression if the omitted variables are not correlated with the other predictors. The estimates from a logistic regression do not behave in the same way (Mood 2010$^{[1]}$). I must confess that this result was quite surprising to me, although this problem seems to be well known$^{[3]}$. Hernán et al. (2011)$^{[4]}$ call this a "mathematical oddity" instead of a bias.
My questions:
- If odds ratios are basically uncomparable across studies and models, how can we combine the results of different studies for binary outcomes?
- What can be said about the countless meta-analyses that did combine the odds ratios from different studies where each study possibly adjusted for a different set of covariates? Are they essentially useless?
References
[1]: Mood, C. (2010). Logistic regression: Why we cannot do what we think we can do, and what we can do about it. European sociological review, 26(1), 67-82.
[2]: Norton EC, Dowd BE, Maciejewski ML (2018): Odds Ratios - Current Best Practice and Use. JAMA 320(1): 84-85.
[3]: Norton EC, Dowd BE (2017): Log Odds and the Interpretation of Logit Models. Health Serv Res. 53(2): 859-878.
[4]: Hernán MA, Clayton D, Keiding N (2011): The Simpson's paradox unraveled. Int J Epidemiol 40: 780-785.
Disclosure
The question (including the R code) is a modified version of a question posed by the user timdisher on datamethods.
R code
set.seed(142857)
n_sims <- 1000 # number of simulations
out <- data.frame(
treat_1 = rep(NA, n_sims)
, treat_2 = rep(NA, n_sims)
, treat_3 = rep(NA, n_sims)
, treat_4 = rep(NA, n_sims)
)
n <- 1000 # number of observations in each simulation
coef_sim <- "x1" # Coefficient of interest
# Coefficients (log-odds)
b0 <- 1
b1 <- log(2)
b2 <- log(2.5)
b3 <- log(3)
b4 <- 0
for(i in 1:n_sims){
x1 <- rbinom(n, 1, 0.5)
x2 <- rnorm(n)
x3 <- rnorm(n)
x4 <- rnorm(n)
z <- b0 + b1*x1 + b2*x2 + b3*x3 + b4*x4
pr <- 1/(1 + exp(-z))
y <- rbinom(n, 1, pr)
df <- data.frame(y = y, x1 = x1, x2 = x2, x3 = x3, x4 = x4)
model1 <- glm(y ~ x1, data = df, family = "binomial")
model2 <- glm(y ~ x1 + x2, data = df, family = "binomial")
model3 <- glm(y ~ x1 + x2 + x3, data = df, family = "binomial")
model4 <- glm(y ~ x1 + x2 + x3 + x4, data = df, family = "binomial")
out$treat_1[i] <- model1$coefficients[coef_sim]
out$treat_2[i] <- model2$coefficients[coef_sim]
out$treat_3[i] <- model3$coefficients[coef_sim]
out$treat_4[i] <- model4$coefficients[coef_sim]
}
# Coefficients
colMeans(out)
exp(colMeans(out)) # Odds ratios
```
