4
$\begingroup$

I am running a logit model on survey data using the svyglm function in the survey package in R. I am using the margins function to find the average marginal effects, but then something weird happens. The confidence intervals for the 's1_022' variable have confidence intervals that do not contain 0, yet the reported p-value is higher than 0.05.

> summary(margins(model1, design=svyvis1))
         factor     AME     SE       z      p   lower   upper
          esup1 -0.1332 0.0155 -8.5863 0.0000 -0.1690 -0.1082
       export_g -0.0022 0.0217 -0.1018 0.9189 -0.0448  0.0402
       import_g -0.0051 0.0133 -0.3830 0.7018 -0.0314  0.0208
         s1_022 -0.0278 0.0147 -1.8938 0.0583 -0.0578 -0.0003
         s1_03a -0.0010 0.0007 -1.5805 0.1140 -0.0024  0.0002
         s1_101  0.0209 0.0144  1.4543 0.1459 -0.0064  0.0500
         s2_202 -0.1031 0.0230 -4.4755 0.0000 -0.1521 -0.0619
         s2_203 -0.0063 0.0290 -0.2189 0.8267 -0.0634  0.0503
         s2_204 -0.0230 0.2224 -0.1033 0.9177 -0.4597  0.4122
         s2_205 -0.0595 0.0296 -2.0074 0.0447 -0.1197 -0.0035
        s2_36a2  0.2657 0.0169 15.7025 0.0000  0.2383  0.3046
         tothrs  0.0002 0.0005  0.4866 0.6265 -0.0007  0.0012
        trim227 -0.0147 0.0252 -0.5834 0.5596 -0.0648  0.0341
        trim228  0.0066 0.0269  0.2465 0.8053 -0.0458  0.0596
        trim229 -0.0645 0.0261 -2.4755 0.0133 -0.1185 -0.0163
        trim230 -0.0129 0.0244 -0.5309 0.5955 -0.0613  0.0343
        trim231 -0.0398 0.0265 -1.5025 0.1330 -0.0934  0.0104
        yprilab -0.0008 0.0003 -2.3803 0.0173 -0.0014 -0.0002

As far as I know, this should not happen since the fact that the confidence intervals do not include zero means that the average marginal effect is not significant. Am I getting something wrong? In which circumstances could this happen?

Improve this question
$\endgroup$
8
  • 4
    $\begingroup$ The confidence intervals are not symmetric about the AME. When this is the case, this result is possible. The CI is not simply $AME \pm c \times SE$. So they are arrived at differently. The p-values are based on the Wald z-statistic, the CI are not Wald CI. Different assumptions, different results. This is assuming the package is not making a programming error. $\endgroup$
    – Heteroskedastic Jim
    Commented Oct 18, 2018 at 22:02
  • $\begingroup$ Are you sure that the reported confidence intervals are 95%? $\endgroup$
    – St. Inkbug
    Commented Oct 19, 2018 at 5:28
  • 2
    $\begingroup$ I googled the information about svyglm. Its manual does not say what methods are used. So it just likes black box. I will not use it before I get the statistical theory behind the black box. $\endgroup$
    – user158565
    Commented Oct 19, 2018 at 14:28
  • 2
    $\begingroup$ @a_statistician in fact the confidence intervals come from the margins package but that is not explicit about how it estimates them either. I think the OP has to look at the code for enlightenment. $\endgroup$
    – mdewey
    Commented Oct 19, 2018 at 14:51
  • $\begingroup$ @mdewey I do not use R. But I hope I could find the theory that support svyglm. I am very disappointed by the search results. $\endgroup$
    – user158565
    Commented Oct 19, 2018 at 14:56

1 Answer 1

Reset to default
6
$\begingroup$

If confidence and p-values are calculated using different methods, then this situation is possible. The $z$-test you have there is a Wald's $z$-test:

$$z=\frac{AME}{SE}\quad\text{then}\quad p=\min{\Big\{1,2\times\big(1-\Phi(\lvert z\rvert)\big)\Big\}}$$

# in R syntax:
min(1, 2 * (1 - pnorm(abs(AME / SE))))

For the confidence intervals to correspond to the p-value as you want, then one must calculate Wald confidence intervals using:

$$AME\pm\Phi^{-1}(.975)\times SE\approx AME\pm1.96\times SE$$

# in R syntax:
AME + qnorm(.975) * c(-SE, SE)

This will produce confidence intervals that are symmetric about the AME i.e. the CI limits will be equidistant from the AME. In your pasted results, the confidence intervals are clearly not symmetric about the AME.

Hence, the CI and p were calculated using different methods, possibly under different assumptions, there is no reason for them to "agree".

I have no idea how the CIs are calculated here. You should probably email the package creator.

Improve this answer
$\endgroup$
2
  • $\begingroup$ You are right. The methods for computing confidence intervals and p-values must be different. Stata seems to calculate symmetric confidence intervals instead. I could not find how CIs are calculated in the R margins documentation, so if nothing else comes up here, emailing the package creator might be a good idea. Thanks!!! $\endgroup$
    – J.M. Molina
    Commented Oct 19, 2018 at 15:20
  • $\begingroup$ @jojas412 usually, if you accept the answer, then you can tick it as accepted. Also, if you're able to read code easily, it might be faster to explore the source code yourself here: github.com/leeper/margins $\endgroup$
    – Heteroskedastic Jim
    Commented Oct 20, 2018 at 4:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.