4
$\begingroup$

After fitting a logit glmer model in R, I got the following coefficient estimate: 

b  = 1.806649
se = 0.9899169
b + qnorm(0.025) * se
b + qnorm(0.05)  * se    # > 0, significant at 90%

I want to report odds ratio (OR) instead, so I calculate OR, its standard error, and re-do inference like so: 

or    = exp(b)
or.se = exp(b) * se      # According to delta method (i.e. one-step Taylor approximation)
or + qnorm(0.05) * or.se # < 1, so no longer significant at 90%

How should I interpret this? I suppose this is because exp(b) no longer follows a normal distribution. If this is the case, how does software (such as Stata) does hypothesis testing to show whether odds ratio $\neq 1$? Does Stata do inference using odds ratio SE or odds ratio CI (i.e. exponentiated log odds ratio CI)?

Improve this question
$\endgroup$

3 Answers 3

Reset to default
5
$\begingroup$

An index having an asymmetric distribution (e.g., a ratio) is generally not one in which a standard error has much meaning. Thus a hypothesis test based on a Wald statistic (of the form estimate minus hypothesized value divided by standard error) is not going to perform well, and confidence interval accuracy is especially affected. Frequentist statisticians tend to favor the likelihood ratio test because it is invariant to transformation of the parameter being tested. If not using a LRT, it is important to use a normality-based test on the right basis such as log odds ratio. Even that has its own problems because even though the log odds ratio is much more normally distributed than the odds ratio, it is still not extremely close to a normal distribution.

Improve this answer
$\endgroup$
2
  • $\begingroup$ I have seen the statement that the odds ratio does not have an asymmetric distribution. 1) Is there a further reading on this? 2) This Stata manual does claim that they use SE(OR) to test $H: OR \neq 1$. (Ctrl-F for The latter test would use the SE(ORb) from the delta rule). Is that possible? $\endgroup$
    – Heisenberg
    Commented Apr 6, 2015 at 17:35
  • 3
    $\begingroup$ It is asymmetric by definition because OR cannot be negative but it has no limit on the high end. $\endgroup$
    – Frank Harrell
    Commented Apr 6, 2015 at 17:44
1
$\begingroup$

I have never had this problem (different p-value for b and OR) in either SAS or Stata. The p-values of b and OR have always been equal for me. The only time I would get different p-values is when I ask for the marginal effects.

Stata example using the 1978 automobile data that comes with the package:

Coefficients

. webuse auto
(1978 Automobile Data)

. logit foreign price mpg weight

Iteration 0:   log likelihood =  -45.03321  
Iteration 1:   log likelihood = -22.244792  
Iteration 2:   log likelihood = -18.069284  
Iteration 3:   log likelihood = -17.184699  
Iteration 4:   log likelihood = -17.161975  
Iteration 5:   log likelihood = -17.161893  
Iteration 6:   log likelihood = -17.161893  

Logistic regression                               Number of obs   =         74
                                                  LR chi2(3)      =      55.74
                                                  Prob > chi2     =     0.0000
Log likelihood = -17.161893                       Pseudo R2       =     0.6189

------------------------------------------------------------------------------
     foreign |      Coef.   Std. Err.      z    P>|z|     [95% Conf. Interval]
-------------+----------------------------------------------------------------
       price |   .0009264   .0003074     3.01   0.003      .000324    .0015288
         mpg |  -.1210918   .0956855    -1.27   0.206     -.308632    .0664483
      weight |  -.0068497   .0019996    -3.43   0.001    -.0107688   -.0029306
       _cons |   14.42237   5.414367     2.66   0.008      3.81041    25.03434
------------------------------------------------------------------------------

Odds ratios

. logit foreign price mpg weight,or

Iteration 0:   log likelihood =  -45.03321  
Iteration 1:   log likelihood = -22.244792  
Iteration 2:   log likelihood = -18.069284  
Iteration 3:   log likelihood = -17.184699  
Iteration 4:   log likelihood = -17.161975  
Iteration 5:   log likelihood = -17.161893  
Iteration 6:   log likelihood = -17.161893  

Logistic regression                               Number of obs   =         74
                                                  LR chi2(3)      =      55.74
                                                  Prob > chi2     =     0.0000
Log likelihood = -17.161893                       Pseudo R2       =     0.6189

------------------------------------------------------------------------------
     foreign | Odds Ratio   Std. Err.      z    P>|z|     [95% Conf. Interval]
-------------+----------------------------------------------------------------
       price |   1.000927   .0003077     3.01   0.003     1.000324     1.00153
         mpg |   .8859526   .0847728    -1.27   0.206      .734451    1.068706
      weight |   .9931737   .0019859    -3.43   0.001     .9892889    .9970737
       _cons |    1834670    9933575     2.66   0.008     45.16896    7.45e+10
------------------------------------------------------------------------------

Marginal effects

. margins,dydx(*)

Average marginal effects                          Number of obs   =         74
Model VCE    : OIM

Expression   : Pr(foreign), predict()
dy/dx w.r.t. : price mpg weight

------------------------------------------------------------------------------
             |            Delta-method
             |      dy/dx   Std. Err.      z    P>|z|     [95% Conf. Interval]
-------------+----------------------------------------------------------------
       price |   .0000686   .0000136     5.04   0.000     .0000419    .0000952
         mpg |  -.0089607    .006596    -1.36   0.174    -.0218886    .0039672
      weight |  -.0005069    .000055    -9.21   0.000    -.0006148    -.000399
------------------------------------------------------------------------------

.

As you can see, only the p-values of the marginal effects are different from the coefficients (but in this example, the conclusion does not change)

Improve this answer
$\endgroup$
4
  • $\begingroup$ Thank you for your help. My question is more about how these softwares calculate the standard error and p-value under the hood $\endgroup$
    – Heisenberg
    Commented Apr 6, 2015 at 6:55
  • $\begingroup$ @AndyW It looks similar, but mpg's standard error is different for log odds ratio and odds ratio. $\endgroup$
    – Heisenberg
    Commented Apr 6, 2015 at 19:30
  • 1
    $\begingroup$ @Heisenberg, the standard errors are different for log odds and OR in the output I posted because they have been exponentiated for the OR. I have always assumed that significance was determined using LR test in Stata for the log odds. I think that is what Maarten mentioned above? $\endgroup$
    – Marquis de Carabas
    Commented Apr 6, 2015 at 23:25
  • $\begingroup$ It seems like the inference with odds ratio is simply copied from the log odds ratio (note how the z score is exactly the same), whereas the odds ratio SE is calculated with delta method. I was misled by a Stata page that claims odds ratio inference is done using odds ratio SE> $\endgroup$
    – Heisenberg
    Commented Apr 7, 2015 at 18:21
1
$\begingroup$

I don't know how Stata works, but I doubt it is doing anything like what you are doing. It is more common to use the likelihood as @FrankHarrell notes. It may help you to read my answer here: Why do my p-values differ between logistic regression output, chi-squared test, and the confidence interval for the OR?

More specific to your question, how good an approximation a Taylor expansion with a given finite number of steps is depends on (a) the function being approximated, and (b) what you are willing to consider 'good'. For this case, looking at the values of the odds ratio at +1 SE, the delta method's approximation does not seem very good to me: 

or + or.se  # [1] 12.11861
exp(b +se)  # [1] 16.38827
Improve this answer
$\endgroup$
7
  • $\begingroup$ I agree that inference for GLM should use the LRT -- but my advisor wants me to report substantive effect instead of log odd ratio. Thus, I'm calculating OR and its std error even though the std error is not that meaningful. Is there a better approach to reporting "substantive effect" of logistic regression? $\endgroup$
    – Heisenberg
    Commented Apr 6, 2015 at 17:42
  • 3
    $\begingroup$ @Heisenberg, I don't have a problem w/ using the odds ratio to report the substantive effect, & I think calculating a CI for the OR is a good thing to do. I would simply profile the likelihood to get the confidence limits. If you prefer a Wald CI for some reason, calculate the Wald CI on the log odds scale & exponentiate each limit to get the limits for the OR. $\endgroup$
    – gung - Reinstate Monica
    Commented Apr 6, 2015 at 17:46
  • $\begingroup$ Calculating the CI using the log odds ratio and exponentiating the bounds is exactly what Stata does by default: stata.com/support/faqs/statistics/delta-rule/index.html $\endgroup$
    – Maarten Buis
    Commented Apr 6, 2015 at 18:51
  • $\begingroup$ Thanks for the tip, @MaartenBuis. That implies Stata is using Wald CIs. Is there a trick to get CIs based on the profiled likelihood? $\endgroup$
    – gung - Reinstate Monica
    Commented Apr 6, 2015 at 19:06
  • 1
    $\begingroup$ @Heisenberg the $p$-values reported in the logistic regression output are for the test that log odds ratio = 0. Given the invariance prorperty of the ML-estimator, this is equivalent to the hypothesis that the odds ratio = 1. However, the sampling distribution is likely to be somewhat better behaved. I have done some simulations, where I have found these tests to be unproblematic. But I mainly work with datasets with about 5 to 10 thousand observations, and my simulations were geared towards such cases. If you have a much smaller sample, then you need to be more careful. $\endgroup$
    – Maarten Buis
    Commented Apr 7, 2015 at 9:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.