If I have one non-significant factor level in a glm, is that entire variable now considered non significant?

Question

I have a question similar to this one, but I just wanted to follow on and ask if the entire variable is now insignificant? I have a factor with 3 levels. When doing the model simplification, it showed that two of the levels were significant, and one was not (p = 0.5). Can I still consider the other two levels to have a significant effect on my response variable, or is that entire variable now non significant? (I expect the insignificant level was insignificant since the sample size for that level was only about 3) Thank you!

Answer 1

When you have a factor variable in a regression model, that is, a categorical variable with multiple levels, that variable should be treated as a whole. So you should mostly disregard the t-tests for each separate level, and test the variable as a whole. In a linear regression model that would be an F-test. Such multiple-df tests are often called chunk tests, see What are chunk tests?.

Most of these is explained in the post you linked. The post I linked above contains examples.

As pointed out in a comment by user Frank Harrell, one specific reason to ignore the 1-df tests is tat they depend on the categorical encoding used. That is, they test specific contrasts, which might well be without any interest to you, especially if you only used some default coding. See my answer at categorical independent variable with three levels and binary logistic regression

Answer 2

Let's consider an example.

We have $50$ dogs, $50$ cats, and $50$ yaks. On some variable of interest, $Y$, dogs follow a $N(0, 1)$ distribution, cats follow a $N(1, 1)$ distribution, and yaks follow a $N(1, 1)$ distribution.

Thus, yaks and cats have the same mean, but dogs have a different mean. When you ask if the means for all three species are the same, the answer is that they definitely are not. Let's run the ANOVA.

set.seed(2021)
N <- 50
y_dog <- rnorm(N, 0, 1)
y_cat <- rnorm(N, 1, 1)
y_yak <- rnorm(N, 1, 1)
y <- c(y_dog, y_cat, y_yak)
species <- c(rep("Dog", N), rep("Cat", N), rep("Yak", N))
      
L <- lm(y ~ species)
summary(L)

Call:
lm(formula = y ~ species)

Residuals:
     Min       1Q   Median       3Q      Max 
-2.59040 -0.66448  0.08034  0.64460  2.56390 

Coefficients:
            Estimate Std. Error t value Pr(>|t|)    
(Intercept)   0.6352     0.1502   4.230  4.1e-05 ***
speciesDog   -0.6193     0.2124  -2.916   0.0041 ** 
speciesYak    0.2545     0.2124   1.199   0.2326    
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Residual standard error: 1.062 on 147 degrees of freedom
Multiple R-squared:  0.1086,    Adjusted R-squared:  0.0965 
F-statistic: 8.957 on 2 and 147 DF,  p-value: 0.0002135

The result of this is that Yaks do not have a significantly different $(\alpha = 0.05)$ mean from cats, and we know that yaks and cats have the same mean. However, the ANOVA emphatically rejects the notion that the three species have the same mean, which we know to be false.

Therefore, the species variable is significant.

APPENDIX

The weirder result is when this kind of "chunk test" gives insignificant p-values for each variable individually but a small p-value for the chunk test, demonstrated below.

library(MASS)
set.seed(2021)
N <- 10
X <- MASS::mvrnorm(N, c(0, 0), matrix(c(1, 0.95, 0.95, 1), 2, 2))
y <- X[, 1] + X[, 2] + rnorm(N)
L <- lm(y ~ X[, 1] + X[, 2])
summary(L)

Call:
lm(formula = y ~ X[, 1] + X[, 2])

Residuals:
    Min      1Q  Median      3Q     Max 
-1.3416 -0.7162  0.0408  0.5220  1.5287 

Coefficients:
            Estimate Std. Error t value Pr(>|t|)
(Intercept)  -0.3595     0.3782  -0.951    0.373
X[, 1]        1.0035     1.4528   0.691    0.512
X[, 2]        1.0648     1.0932   0.974    0.362

Residual standard error: 1.105 on 7 degrees of freedom
Multiple R-squared:  0.7986,    Adjusted R-squared:  0.741 
F-statistic: 13.88 on 2 and 7 DF,  p-value: 0.003668

This is not quite what you asked, but I do not want you to think that you need any one parameter in a chunk test to have a significant p-value.