I have a question similar to this one, but I just wanted to follow on and ask if the entire variable is now insignificant? I have a factor with 3 levels. When doing the model simplification, it showed that two of the levels were significant, and one was not (p = 0.5). Can I still consider the other two levels to have a significant effect on my response variable, or is that entire variable now non significant? (I expect the insignificant level was insignificant since the sample size for that level was only about 3) Thank you!
When you have a factor variable in a regression model, that is, a categorical variable with multiple levels, that variable should be treated as a whole. So you should mostly disregard the t-tests for each separate level, and test the variable as a whole. In a linear regression model that would be an F-test. Such multiple-df tests are often called chunk tests, see What are chunk tests?.
Most of these is explained in the post you linked. The post I linked above contains examples.
As pointed out in a comment by user Frank Harrell, one specific reason to ignore the 1-df tests is tat they depend on the categorical encoding used. That is, they test specific contrasts, which might well be without any interest to you, especially if you only used some default coding. See my answer at categorical independent variable with three levels and binary logistic regression
Let's consider an example.
We have $50$ dogs, $50$ cats, and $50$ yaks. On some variable of interest, $Y$, dogs follow a $N(0, 1)$ distribution, cats follow a $N(1, 1)$ distribution, and yaks follow a $N(1, 1)$ distribution.
Thus, yaks and cats have the same mean, but dogs have a different mean. When you ask if the means for all three species are the same, the answer is that they definitely are not. Let's run the ANOVA.
set.seed(2021) N <- 50 y_dog <- rnorm(N, 0, 1) y_cat <- rnorm(N, 1, 1) y_yak <- rnorm(N, 1, 1) y <- c(y_dog, y_cat, y_yak) species <- c(rep("Dog", N), rep("Cat", N), rep("Yak", N)) L <- lm(y ~ species) summary(L)
Call: lm(formula = y ~ species) Residuals: Min 1Q Median 3Q Max -2.59040 -0.66448 0.08034 0.64460 2.56390 Coefficients: Estimate Std. Error t value Pr(>|t|) (Intercept) 0.6352 0.1502 4.230 4.1e-05 *** speciesDog -0.6193 0.2124 -2.916 0.0041 ** speciesYak 0.2545 0.2124 1.199 0.2326 --- Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1 Residual standard error: 1.062 on 147 degrees of freedom Multiple R-squared: 0.1086, Adjusted R-squared: 0.0965 F-statistic: 8.957 on 2 and 147 DF, p-value: 0.0002135
The result of this is that Yaks do not have a significantly different $(\alpha = 0.05)$ mean from cats, and we know that yaks and cats have the same mean. However, the ANOVA emphatically rejects the notion that the three species have the same mean, which we know to be false.
Therefore, the species variable is significant.
The weirder result is when this kind of "chunk test" gives insignificant p-values for each variable individually but a small p-value for the chunk test, demonstrated below.
library(MASS) set.seed(2021) N <- 10 X <- MASS::mvrnorm(N, c(0, 0), matrix(c(1, 0.95, 0.95, 1), 2, 2)) y <- X[, 1] + X[, 2] + rnorm(N) L <- lm(y ~ X[, 1] + X[, 2]) summary(L)
Call: lm(formula = y ~ X[, 1] + X[, 2]) Residuals: Min 1Q Median 3Q Max -1.3416 -0.7162 0.0408 0.5220 1.5287 Coefficients: Estimate Std. Error t value Pr(>|t|) (Intercept) -0.3595 0.3782 -0.951 0.373 X[, 1] 1.0035 1.4528 0.691 0.512 X[, 2] 1.0648 1.0932 0.974 0.362 Residual standard error: 1.105 on 7 degrees of freedom Multiple R-squared: 0.7986, Adjusted R-squared: 0.741 F-statistic: 13.88 on 2 and 7 DF, p-value: 0.003668
This is not quite what you asked, but I do not want you to think that you need any one parameter in a chunk test to have a significant p-value.