2
$\begingroup$

I am running logistic regression models to compare the impact of different indicators using Stata. As these comparisons may lead to false conclusion due to confounding and rescaling if log-odds or odds ratios are compared (see Karlson/Holm/Breen 2012 and Mood 2010), I am comparing average marginal effects. However, Stata sometimes shows z- and p-values which differ between the original log-odds estimation and the average marginal effects estimation of the same model.
My question is, which of these different z- and p-values should I include or use in a table which compares the models, e.g. to print the popular significance stars?

Here is an example, which shows the existence of these estimation differences with example data from Stata:

. webuse acmemanuf

. logit acceptable i.temp y, nolog

Logistic regression                               Number of obs   =         49
                                                  LR chi2(3)      =      14.43
                                                  Prob > chi2     =     0.0024
Log likelihood = -23.740177                       Pseudo R2       =     0.2330

------------------------------------------------------------------------------
  acceptable |      Coef.   Std. Err.      z    P>|z|     [95% Conf. Interval]
-------------+----------------------------------------------------------------
        temp |
          2  |   .7382167   .8717338     0.85   0.397    -.9703501    2.446784
          3  |   .6142972   .9916756     0.62   0.536    -1.329351    2.557946
             |
           y |   .1053559   .0382794     2.75   0.006     .0303297    .1803821
       _cons |  -10.87168   4.134835    -2.63   0.009    -18.97581   -2.767554
------------------------------------------------------------------------------

. margins, dydx(*)

Average marginal effects                          Number of obs   =         49
Model VCE    : OIM

Expression   : Pr(acceptable), predict()
dy/dx w.r.t. : 2.temp 3.temp y

------------------------------------------------------------------------------
             |            Delta-method
             |      dy/dx   Std. Err.      z    P>|z|     [95% Conf. Interval]
-------------+----------------------------------------------------------------
        temp |
          2  |    .122275   .1421648     0.86   0.390    -.1563629    .4009129
          3  |   .1028261   .1626389     0.63   0.527    -.2159403    .4215925
             |
           y |   .0171374   .0041764     4.10   0.000     .0089518     .025323
------------------------------------------------------------------------------
Note: dy/dx for factor levels is the discrete change from the base level.

The differences between the estimations for the two "temperature" values are very small and may even occur due to rounding issues etc., but the differences for the "y"-variable are relatively large, especially if one considers the z-values. Of course, it does not matter much in this case as no p-value falls under or climbs above conventional limits. But what if they would do so?

Literature:
Karlson, K.B., A. Holm & R. Breen, 2012: Comparing Regression Coefficients Between Same-sample Nested Models Using Logit and Probit: A New Method. Sociological Methodology 42: 286–313.
Mood, C., 2010: Logistic Regression: Why We Cannot Do What We Think We Can Do, and What We Can Do About It. European Sociological Review 26: 67–82.

$\endgroup$
2
$\begingroup$

I would report the p-values from marginal effects since that is what you ultimately care about. The index function coefficients are not very meaningful in their raw form.

Generally, unless we are dealing with interactions, the sign and significance of the index function coefficients track the corresponding quantities of the various marginal effects pretty closely. That is still the case here.

As a practical matter, you can use the post option in margins to make these available to estout or outreg in making your tables.

$\endgroup$
  • $\begingroup$ If I use the post option, I am loosing many of the summary statistics. Do you happen to know whether there is a way, to add the z-statistics from the margins post-estimation using estadd instead of post as described in the documentation. They are stored in the e(margins_table) together with all other estimates. Doing esttab, main(margins_b) margin still gives me the wrong t-statistics from the original estimation after running estadd margins, dydx(*). $\endgroup$ – non-numeric_argument Oct 30 '14 at 9:31
  • $\begingroup$ Please, note that I asked the same question from the comment on Stackoverflow. $\endgroup$ – non-numeric_argument Oct 30 '14 at 9:44
  • 1
    $\begingroup$ @non-numeric_argument I answered this on SO. $\endgroup$ – Dimitriy V. Masterov Oct 30 '14 at 17:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.