# Why does hypothesis testing using coefficient and odds ratio give different conclusion?

After fitting a logit glmer model in R, I got the following coefficient estimate:

b  = 1.806649
se = 0.9899169
b + qnorm(0.025) * se
b + qnorm(0.05)  * se    # > 0, significant at 90%


I want to report odds ratio (OR) instead, so I calculate OR, its standard error, and re-do inference like so:

or    = exp(b)
or.se = exp(b) * se      # According to delta method (i.e. one-step Taylor approximation)
or + qnorm(0.05) * or.se # < 1, so no longer significant at 90%


How should I interpret this? I suppose this is because exp(b) no longer follows a normal distribution. If this is the case, how does software (such as Stata) does hypothesis testing to show whether odds ratio $\neq 1$? Does Stata do inference using odds ratio SE or odds ratio CI (i.e. exponentiated log odds ratio CI)?

An index having an asymmetric distribution (e.g., a ratio) is generally not one in which a standard error has much meaning. Thus a hypothesis test based on a Wald statistic (of the form estimate minus hypothesized value divided by standard error) is not going to perform well, and confidence interval accuracy is especially affected. Frequentist statisticians tend to favor the likelihood ratio test because it is invariant to transformation of the parameter being tested. If not using a LRT, it is important to use a normality-based test on the right basis such as log odds ratio. Even that has its own problems because even though the log odds ratio is much more normally distributed than the odds ratio, it is still not extremely close to a normal distribution.

• I have seen the statement that the odds ratio does not have an asymmetric distribution. 1) Is there a further reading on this? 2) This Stata manual does claim that they use SE(OR) to test $H: OR \neq 1$. (Ctrl-F for The latter test would use the SE(ORb) from the delta rule). Is that possible? – Heisenberg Apr 6 '15 at 17:35
• It is asymmetric by definition because OR cannot be negative but it has no limit on the high end. – Frank Harrell Apr 6 '15 at 17:44

I have never had this problem (different p-value for b and OR) in either SAS or Stata. The p-values of b and OR have always been equal for me. The only time I would get different p-values is when I ask for the marginal effects.

Stata example using the 1978 automobile data that comes with the package:

Coefficients

. webuse auto
(1978 Automobile Data)

. logit foreign price mpg weight

Iteration 0:   log likelihood =  -45.03321
Iteration 1:   log likelihood = -22.244792
Iteration 2:   log likelihood = -18.069284
Iteration 3:   log likelihood = -17.184699
Iteration 4:   log likelihood = -17.161975
Iteration 5:   log likelihood = -17.161893
Iteration 6:   log likelihood = -17.161893

Logistic regression                               Number of obs   =         74
LR chi2(3)      =      55.74
Prob > chi2     =     0.0000
Log likelihood = -17.161893                       Pseudo R2       =     0.6189

------------------------------------------------------------------------------
foreign |      Coef.   Std. Err.      z    P>|z|     [95% Conf. Interval]
-------------+----------------------------------------------------------------
price |   .0009264   .0003074     3.01   0.003      .000324    .0015288
mpg |  -.1210918   .0956855    -1.27   0.206     -.308632    .0664483
weight |  -.0068497   .0019996    -3.43   0.001    -.0107688   -.0029306
_cons |   14.42237   5.414367     2.66   0.008      3.81041    25.03434
------------------------------------------------------------------------------


Odds ratios

. logit foreign price mpg weight,or

Iteration 0:   log likelihood =  -45.03321
Iteration 1:   log likelihood = -22.244792
Iteration 2:   log likelihood = -18.069284
Iteration 3:   log likelihood = -17.184699
Iteration 4:   log likelihood = -17.161975
Iteration 5:   log likelihood = -17.161893
Iteration 6:   log likelihood = -17.161893

Logistic regression                               Number of obs   =         74
LR chi2(3)      =      55.74
Prob > chi2     =     0.0000
Log likelihood = -17.161893                       Pseudo R2       =     0.6189

------------------------------------------------------------------------------
foreign | Odds Ratio   Std. Err.      z    P>|z|     [95% Conf. Interval]
-------------+----------------------------------------------------------------
price |   1.000927   .0003077     3.01   0.003     1.000324     1.00153
mpg |   .8859526   .0847728    -1.27   0.206      .734451    1.068706
weight |   .9931737   .0019859    -3.43   0.001     .9892889    .9970737
_cons |    1834670    9933575     2.66   0.008     45.16896    7.45e+10
------------------------------------------------------------------------------


Marginal effects

. margins,dydx(*)

Average marginal effects                          Number of obs   =         74
Model VCE    : OIM

Expression   : Pr(foreign), predict()
dy/dx w.r.t. : price mpg weight

------------------------------------------------------------------------------
|            Delta-method
|      dy/dx   Std. Err.      z    P>|z|     [95% Conf. Interval]
-------------+----------------------------------------------------------------
price |   .0000686   .0000136     5.04   0.000     .0000419    .0000952
mpg |  -.0089607    .006596    -1.36   0.174    -.0218886    .0039672
weight |  -.0005069    .000055    -9.21   0.000    -.0006148    -.000399
------------------------------------------------------------------------------


.

As you can see, only the p-values of the marginal effects are different from the coefficients (but in this example, the conclusion does not change)

• Thank you for your help. My question is more about how these softwares calculate the standard error and p-value under the hood – Heisenberg Apr 6 '15 at 6:55
• @AndyW It looks similar, but mpg's standard error is different for log odds ratio and odds ratio. – Heisenberg Apr 6 '15 at 19:30
• @Heisenberg, the standard errors are different for log odds and OR in the output I posted because they have been exponentiated for the OR. I have always assumed that significance was determined using LR test in Stata for the log odds. I think that is what Maarten mentioned above? – Marquis de Carabas Apr 6 '15 at 23:25
• It seems like the inference with odds ratio is simply copied from the log odds ratio (note how the z score is exactly the same), whereas the odds ratio SE is calculated with delta method. I was misled by a Stata page that claims odds ratio inference is done using odds ratio SE> – Heisenberg Apr 7 '15 at 18:21

I don't know how Stata works, but I doubt it is doing anything like what you are doing. It is more common to use the likelihood as @FrankHarrell notes. It may help you to read my answer here: Why do my p-values differ between logistic regression output, chi-squared test, and the confidence interval for the OR?

More specific to your question, how good an approximation a Taylor expansion with a given finite number of steps is depends on (a) the function being approximated, and (b) what you are willing to consider 'good'. For this case, looking at the values of the odds ratio at +1 SE, the delta method's approximation does not seem very good to me:

or + or.se  # [1] 12.11861
exp(b +se)  # [1] 16.38827

• I agree that inference for GLM should use the LRT -- but my advisor wants me to report substantive effect instead of log odd ratio. Thus, I'm calculating OR and its std error even though the std error is not that meaningful. Is there a better approach to reporting "substantive effect" of logistic regression? – Heisenberg Apr 6 '15 at 17:42
• @Heisenberg, I don't have a problem w/ using the odds ratio to report the substantive effect, & I think calculating a CI for the OR is a good thing to do. I would simply profile the likelihood to get the confidence limits. If you prefer a Wald CI for some reason, calculate the Wald CI on the log odds scale & exponentiate each limit to get the limits for the OR. – gung - Reinstate Monica Apr 6 '15 at 17:46
• Calculating the CI using the log odds ratio and exponentiating the bounds is exactly what Stata does by default: stata.com/support/faqs/statistics/delta-rule/index.html – Maarten Buis Apr 6 '15 at 18:51
• Thanks for the tip, @MaartenBuis. That implies Stata is using Wald CIs. Is there a trick to get CIs based on the profiled likelihood? – gung - Reinstate Monica Apr 6 '15 at 19:06
• @Heisenberg the $p$-values reported in the logistic regression output are for the test that log odds ratio = 0. Given the invariance prorperty of the ML-estimator, this is equivalent to the hypothesis that the odds ratio = 1. However, the sampling distribution is likely to be somewhat better behaved. I have done some simulations, where I have found these tests to be unproblematic. But I mainly work with datasets with about 5 to 10 thousand observations, and my simulations were geared towards such cases. If you have a much smaller sample, then you need to be more careful. – Maarten Buis Apr 7 '15 at 9:36