# Model interpretation in R (anova vs summary output)

I have a question concerning model selection. I am comparing three models where at each step I added a new term as both fixed effect and as random slope:

baseline <- lme(like_rating ~ 1,
random = list(~ 1 | PID, ~1| stimulusID),
control=ctrl,
method = "ML", data = DataScaled)

mfit_1 <- lme(like_rating ~ V1,
random = list(~ V1 | PID, ~1| stimulusID),
control=ctrl,
method = "ML", data = DataScaled)

mfit_2 <-lme(like_rating ~ V1 + V2,
random = list(~ V1 + V2 | PID, ~1| stimulusID),
control=ctrl,
method = "ML", data = DataScaled)


To select the best model, I then compare the new models with the baseline using the anova function, and obtained the results below:

anova(baseline, mfit1, mfit2)
#             Model df      AIC      BIC    logLik   Test  L.Ratio p-value
# baseline        1  4 16976.72 16999.06 -8484.359
# V1              2 11 16972.95 17034.38 -8475.475 1 vs 2 17.76746  0.0131
# V2              3 16 16966.74 17056.09 -8467.368 2 vs 3 16.21444  0.0063


Above it looks like all the variables significantly improved the model. However, when I call the summary(mfit_2) function for the latest saturated model, the output shows that V2 is NS:

# Fixed effects: like_rating ~ Contour + jpgSiz
#                 Value Std.Error   DF   t-value p-value
# (Intercept)  43.29319 2.7998348 1924 15.462767  0.0000
# V1_levelA    -2.11111 0.9833151 1924 -2.146931  0.0319
# V1_levelB    -3.28676 1.1867284 1924 -2.769596  0.0057
# V2           -0.96441 0.4814861 1924 -2.002988  0.3093


EDIT: Question: How should I interpret the contribution of V2 in my model?

Insights on this would be valuable and much appreciated.

• How the column L.Ratio is calculated from column logLik? I cannot use numbers in logLik to obtain values in L.Ratio. Dec 13, 2020 at 22:27

The anova() and summary() perform two different tests. The first is clearly an ANOVA test, while summary simple performs a t-test. T-tests should not be used to compare the explanatory capabilities of two models, because the t-test was set up just to the check significance of a predictor to be different from 0. In stepwise selection it is common to have at some point a variable that is not statistically significant accordirg to a t-test, but after some variables have been added or removed it may become significant. That's due to the selection procedure. Ideally to find the best subset of variables we should fit all the possible models and compare them, but that is too computationally expansive. On the other hand ANOVA test aims to detect whether the difference of explanatory power between two models is significant. If it is not, we prefer the model with fewer variables, because it is more parsimonious (Occam razor). So in your case you should select the second model, the one with V2. By the way instead of looking to the t-tests I suggest you to check AIC and BIC indexes, they give you a measure of the model goodness. The higher they are the worse they are, so in your case AIC decreases and it is consistent with what ANOVA test is showing, while BIC has increased. It depends on a how they are defined, and it is not so rare that they do not agree.