Inconsistent P-Value and Wald Confidence Interval in R Survey Package

Question

I am using the svyglm function in the survey package in R to fit logistic regression models to a stratified, cluster survey. I want to calculate confidence intervals for my regression coefficients. The default method for confint.svyglm says that it creates Wald confidence intervals by adding and subtracting a multiple of the standard error. But the confidence interval this produces is not consistent with the p-value from the model - confidence intervals that do not overlap 0 still have p-values greater than .05.

I tried to replicate the p-value and confidence interval calculations by hand. It appears the p-value is calculated using a t-test, with the df of the t distribution taken from the residual degrees of freedom from the model. So far so good. But the confidence interval provided by confint.svyglm is just coefficient +/- 1.96*standard.error. This seems wrong - for a 95% confidence interval, I think the multiplier for the standard error should be the .975 quantile of a t-distribution with the appropriate degrees of freedom (in my case 10), which can be somewhat different from 1.96 (the .975 quantile of a z-distribution). True? Has anyone else had this problem? I am relatively new to working with survey data. Is there a reason to always use the z-quantile instead of the t-quantile for complex surveys specifically, or is this just a bug in the package?

Answer 1

If you are fitting a logistic regression model, the parameters are usually reported as odds ratios. One can show that the odds ratio for independent variable $i$ is given by $e^{\beta_i}$. Thus, if $\beta_i=0$, then $e^{\beta_i}=e^0=1$. An odds ratio of 1 indicates no effect of that independent variable on the dependent variable, just like a coefficient of 0 would indicate no effect of that independent variable on the dependent variable in a linear regression model. Thus for any estimate where $p>0.05$, your confidence interval should contain 1; whether or not the interval contains 0 is of no relevance when assessing the significance of a predictor in a logistic regression model.

Confidence intervals for $\beta_i$ are of the form $\beta_i\pm t^*_{df}\frac{s}{\sqrt{n}}$, but if your estimates are reported as odds ratios (i.e. $e^{\beta_i}$), then your intervals will likely be reported in the form $e^{\beta_i\pm t^*_{df}\frac{s}{\sqrt{n}}}$. Since we're taking $\beta_i$ and adding or subtracting a constant, then exponentiating the result, your confidence interval is going to be lopsided - that is, your estimate $e^\hat{\beta_i}$ will not be exactly in the middle of your confidence interval.

In order to check whether or not your estimates and intervals are provided in odds ratio form, you should try taking the natural log of each estimate, generating the confidence intervals using $z$ or $t$ (whichever you feel is appropriate), and exponentiate both sides to get your confidence interval.

As for using 1.96 versus the critical $t$-value, you are correct in that it will depend on your sample size. I would err on the side of caution and use the $t$-distribution rather than the $z$-distribution to find the answer. I would imagine that this is simply an issue within the package.