Model interpretation in R (anova vs summary output)

Question

I have a question concerning model selection. I am comparing three models where at each step I added a new term as both fixed effect and as random slope:

baseline <- lme(like_rating ~ 1,
                random = list(~ 1 | PID, ~1| stimulusID),
                control=ctrl,
                method = "ML", data = DataScaled)

mfit_1 <- lme(like_rating ~ V1,
              random = list(~ V1 | PID, ~1| stimulusID),
              control=ctrl,
              method = "ML", data = DataScaled)

mfit_2 <-lme(like_rating ~ V1 + V2,
                  random = list(~ V1 + V2 | PID, ~1| stimulusID),
                  control=ctrl,
                  method = "ML", data = DataScaled)

To select the best model, I then compare the new models with the baseline using the anova function, and obtained the results below:

anova(baseline, mfit1, mfit2)
#             Model df      AIC      BIC    logLik   Test  L.Ratio p-value
# baseline        1  4 16976.72 16999.06 -8484.359                        
# V1              2 11 16972.95 17034.38 -8475.475 1 vs 2 17.76746  0.0131
# V2              3 16 16966.74 17056.09 -8467.368 2 vs 3 16.21444  0.0063

Above it looks like all the variables significantly improved the model. However, when I call the summary(mfit_2) function for the latest saturated model, the output shows that V2 is NS:

# Fixed effects: like_rating ~ Contour + jpgSiz 
#                 Value Std.Error   DF   t-value p-value
# (Intercept)  43.29319 2.7998348 1924 15.462767  0.0000
# V1_levelA    -2.11111 0.9833151 1924 -2.146931  0.0319
# V1_levelB    -3.28676 1.1867284 1924 -2.769596  0.0057
# V2           -0.96441 0.4814861 1924 -2.002988  0.3093

EDIT: Question: How should I interpret the contribution of V2 in my model?

Insights on this would be valuable and much appreciated.

Answer 1

The anova() and summary() perform two different tests. The first is clearly an ANOVA test, while summary simple performs a t-test. T-tests should not be used to compare the explanatory capabilities of two models, because the t-test was set up just to the check significance of a predictor to be different from 0. In stepwise selection it is common to have at some point a variable that is not statistically significant accordirg to a t-test, but after some variables have been added or removed it may become significant. That's due to the selection procedure. Ideally to find the best subset of variables we should fit all the possible models and compare them, but that is too computationally expansive. On the other hand ANOVA test aims to detect whether the difference of explanatory power between two models is significant. If it is not, we prefer the model with fewer variables, because it is more parsimonious (Occam razor). So in your case you should select the second model, the one with V2. By the way instead of looking to the t-tests I suggest you to check AIC and BIC indexes, they give you a measure of the model goodness. The higher they are the worse they are, so in your case AIC decreases and it is consistent with what ANOVA test is showing, while BIC has increased. It depends on a how they are defined, and it is not so rare that they do not agree.

Ok, now I get it. You're right it's quite strange, it's a borderline case. There is no universal choice in statistics, meaning that if you are very convinced (and you are able to prove) that the test is "mistaken" you can take an opposite choice. Notice that is sensible with how the hypothesis test has been defined. In this case I would procede in the following way. Just by considering t-test and ANOVA test, I would select model with V2. But we have also AIC And BIC that do not change much compared to their magnitudes, that is an evidence against the result of the ANOVA test. In this particular case with only these info I'd remove V2. However I suggest you to perform ANOVA using another test statistic, you can find which ones are available on R documentation. Finally I also suggest you to take into account what you think of the problem if you have some domain knowledge, e.g. if you think that there is a casuality relationship between your predictor V2 and your response variable you may keep it.