How to interpret anova(model, test = “Chisq”) for logistic regression model?

Question

I’m new to logistic regression and was hoping for some help in picking a "best fit model". Say I have a group of students who are assigned a job after college. Everyone can request their top 5, but there's no telling if they'll receive it or not.

Say I use logistic regression to see if a student receives their top job choice or not (0=no, 1=yes) and then a variety of independent variables

• gender (Male or Female)
• race (White, Asian, Black, Hispanic, Other)
• math gpa (continuous 0-100)
• English gpa (continuous 0-100)
• tier (top, middle, or bottom of class)

How do I know which variables I should keep in the model for best fit?

So if I run the model with all the variables and I get an output like this

Call:

glm(formula = receive_top1  ~ gender + race + tier + math_gpa + 
       English_gpa, family = "binomial", data = data)
   
Coefficients:   
               Estimate      Std. Error.     Z Value           Pr(>|z|)  
(Intercept)      -16.7.          2.95.         -5.66.       0.00000000015 
genderFemale      0.09           0.19           0.48            0.63416       
raceAsian        -0.26           0.40          -0.66            0.50936 
raceBlack         0.33           0.35           0.94            0.34580 
raceHispanic      0.23           0.26           0.88            0.38112 
raceOther         0.43           0.32           1.35            0.17610  
tierMiddle       -0.13           0.23          -0.55            0.57952 
tierTop          -0.64           0.35          -1.85            0.06433 
math_gpa          0.10           0.02           4.92       0.00000000888 
English_gpa       0.06           0.02           2.80            0.00504 

Null Deviance: 1327.5 on 967 degrees of freedom
Residual Deviance: 1249.6 on 957 degrees of freedom
AIC: 1271.6

And I'm using a p-value of 0.05, how do I know which variables are important to the model? It seems to me like the intercept, math_gpa and English_gpa are significant.

However when I run anova(model, test = "Chisq") I get confused.
Model: binomial, link: logit
Response: receive_top1
Terms added sequentially (first to last)
DF Deviance Resid. DF Resid. Dev Pr(>Chi)
NULL 967 1327.5
gender 1 1.15 966 1326.4 0.2836
race 4 1.39 962 1325.0 0.8467
tier 2 40.44 960 1284.5 0.00000001657
math_gpa 1 23.06 959 1261.5 0.00000015
English_gpa 1 7.96 957 1249.7 0.00478

But now looking at the anova output this makes it seems like tier is also significant as well as math_gpa and English_gpa. Could someone also explain what the null hypothesis is in this case for the anova function?

How do I know what independent variables to keep in my model? Should I keep tier in? Take it out?

Answer 1

Its important to understand what happens in these tests when your model contains categorical variables.

Tier is comprised of 3 categories (Top, Middle, and what I presume is Low). The p values you see in summary correspond to individual Wald tests, but those p values do not tell you if Tier (that is, the entire categorical variable) can explain the probability of getting your first choice. You have to test the coefficients for Tier together, not separately.

To do that, you perform a deviance goodness of fit test. This test is a lot like the F test in linear regression. Essentially, you compare the reduction of deviance between two models: one with and one without Tier. Rejecting the null of this test means the reduction in deviance is larger than one would expect if Tier did not explain anything about the outcome (equivalently, if all coefficients for Tier were really 0). It is important to note that the test you see in the aov call only corresponds to a test between a model containing gender, race, and tier and a model containing only gender and race. Because gpa is found at the bottom of the list, the test does not account for any change in deviance due to gpa. This can be problematic if results are confounded by gpa.

Which brings me to my final point. Model selection for an inference problem is really not a data centric problem. Indeed, any sort of model selection after fitting the full model is kind of like doing stepwise selection (and we all know how much this site likes stepwise). So long as you initally believed that gpa, tier, gender, and race affected the probability of the outcome, there is no need to select variables out of the model. Indeed, doing so might lead to omitted variable bias.