Wald test failing to reject null hypothesis that a predictor is insignificant in regression when t-test on terms involving predictor show significance

Question

I am running a logistic regression using a set of several predictors as well as several interaction terms. For each predictor, I am running a Wald test in statsmodels to test whether or not a predictor is significant in my model. I am doing this by testing the joint hypothesis that all terms involving the predictor (individually and in interaction terms) are zero. In some instances, I am unable to reject this null hypothesis, yet at the same time I can find instances of the predictor's interaction terms individually appearing significant in the result returned from statsmodels' summary method. So a contrived example would be, I check whether or not all coefficients for terms involving x1 are zero using a Wald test and find that I cannot reject the hypothesis that they are zero, but then looking at the interaction terms individually, I find the interaction term x1x2 has a p-value <0.05. How can this be possible? Is it perhaps an indication that the wald_test function I am using is not testing the hypothesis I think it is testing (i.e. I have specified it incorrectly in the r_matrix)?

Answer 1

I check whether or not all coefficients for terms involving x1 are zero using a Wald test and find that I cannot reject the hypothesis that they are zero, but then looking at the interaction terms individually, I find the interaction term x1x2 has a p-value <0.05. How can this be possible?

You might think about this as similar to the problem with multiple comparisons. The more comparisons you make, the more likely that a single comparison might appear to be "significant" by chance. Alternatively, if you do a lot of comparisons most of which aren't "significant" and account appropriately for the multiple comparisons, you can mask a truly "significant" result

A Wald test on multiple coefficients evaluates the quadratic form produced by the coefficient estimates and the inverse of their covariance matrix against a chi-square distribution with degrees of freedom equal to the number of coefficients. Even if one of those coefficients is actually "significant," including multiple insignificant coefficients in the multiple-coefficient test can swamp out the "significance."

As a crude illustration, remember that the chi-square distribution with $k$ degrees of freedom used as the null distribution represents the sum of the squares of $k$ independent standard normal variables. The mean of that distribution is $k$. For a single coefficient you might have a "significant" chi-square statistic of 5:

1-pchisq(5,df=1)
# [1] 0.02534732

but if you add in 3 more standard normal variates contributing a total of 3 (mean under the null) to the combined chi-square statistic in a 4-coefficient Wald test the overall "significance" at p < 0.05 disappears.

1-pchisq(5+3,df=4)
# [1] 0.09157819

That's one reason why it's typically best to have some specific pre-specified hypotheses rather than throwing everything into the model and seeing what turns out to be "significant."