I am currently working with a logistic model in R and I have a question regarding the p-values that I get from different types of tests.
Since I have separation in my data (caused by X) I use the brglm2 package and my models look like this:
model1 = glm(Y ~ X + D + S, family = binomial(link = "logit"), method = "brglm_fit", data = data ) model2 = glm(Y ~ X + D, family = binomial(link = "logit"), method = "brglm_fit", data = data )
Y: 0 or 1
S: factor with 3 levels
D: factor with 2 levels
To test for example for the significance of S, I use:
anova(model1, model2, test="Chisq")
Analysis of Deviance Table Model 1: Y ~ X + D Model 2: Y ~ X + S + D Resid. Df Resid. Dev Df Deviance Pr(>Chi) 1 265 17.0218 2 263 8.1608 2 8.861 0.01191 *
As I understand this is the two models significantly differ from each other and that is the effect of S (or from leaving it out).
If I use the
Anova function of the car package (type II or type III), I get slightly different results (depending on the model they can be totally different)
Anova(model1) Analysis of Deviance Table (Type II tests) Response: Y LR Chisq Df Pr(>Chisq) X 306.718 1 < 2e-16 *** S 8.085 2 0.01756 * D 5.814 1 0.01590 *
summary( glht(model1, linfct=mcp(S = "Tukey")) ) Simultaneous Tests for General Linear Hypotheses Multiple Comparisons of Means: Tukey Contrasts Fit: glm(formula = Y ~ X + S + D, family = binomial, data = data, control = list(maxit = 10000), method = "brglm_fit") Linear Hypotheses: Estimate Std. Error z value Pr(>|z|) K - A == 0 -3.1136 1.9280 -1.615 0.239 R - A == 0 -3.7331 1.9983 -1.868 0.148 R - K == 0 -0.6195 1.7794 -0.348 0.935 (Adjusted p values reported -- single-step method)
I understand that the
anova() function tests the difference between the models and that
Anova() tests for the effects of the variables on the dependent variable with different types of SS and that the results from the two can therefore differ. I usually rely rather on the results obtained from the model comparison. But how does it happen that the post-hoc test is so far off? Is this not the appropriate way to test for the effect of the different levels of a factor in logistic regression?