anova and summry, inconsistent results

Question

I am fitting a linear mixed effect models with two factors (mPair with 6 levels, and spd_des with 3 levels) and their interaction, using "sum" contrasts. The summary of the fit and an anova (using type III sum of squares) provide inconsistent results.

> options(contrasts = c("contr.sum","contr.poly"))
> lCtr  <- lmeControl(maxIter = 1000, niterEM = 500, msVerbose = FALSE, opt = 'optim')
> linM2 <- lme(cc_marg ~ mPair*spd_des , random = ~1|ratID, data=dat_trf, na.action=na.omit, method = "ML", control=lCtr )
> summary(linM2)
Linear mixed-effects model fit by maximum likelihood
 Data: dat_trf 
       AIC      BIC   logLik
  30.12325 86.03906 4.938375

Random effects:
 Formula: ~1 | ratID
        (Intercept)  Residual
StdDev:   0.1006728 0.2210174

Fixed effects: cc_marg ~ mPair * spd_des 
                     Value  Std.Error DF   t-value p-value
(Intercept)      0.8347112 0.04320961 94 19.317719  0.0000
mPair1           0.5809747 0.04549001 94 12.771481  0.0000
mPair2          -0.3207539 0.05823676 94 -5.507757  0.0000
mPair3          -0.1815169 0.04583826 94 -3.959943  0.0001
mPair4           0.0168609 0.05823676 94  0.289523  0.7728
mPair5          -0.1849861 0.04583826 94 -4.035626  0.0001
spd_des1         0.0732885 0.03211197 94  2.282281  0.0247
spd_des2        -0.0253861 0.03227828 94 -0.786477  0.4336
mPair1:spd_des1 -0.0260983 0.06110193 94 -0.427128  0.6703
mPair2:spd_des1  0.0947705 0.07959226 94  1.190699  0.2368
mPair3:spd_des1 -0.0816196 0.06261930 94 -1.303425  0.1956
mPair4:spd_des1  0.0062235 0.07959226 94  0.078193  0.9378
mPair5:spd_des1 -0.0238940 0.06261930 94 -0.381576  0.7036
mPair1:spd_des2  0.0069180 0.06236754 94  0.110923  0.9119
mPair2:spd_des2 -0.0471758 0.07965722 94 -0.592236  0.5551
mPair3:spd_des2  0.0207045 0.06262016 94  0.330636  0.7417
mPair4:spd_des2  0.0302277 0.07965722 94  0.379472  0.7052
mPair5:spd_des2  0.0102634 0.06262016 94  0.163899  0.8702

> anova.lme(linM2,type="marginal")
              numDF denDF  F-value p-value
(Intercept)       1    94 373.1743  <.0001
mPair             5    94  42.1607  <.0001
spd_des           2    94   2.6265  0.0776
mPair:spd_des    10    94   0.3150  0.9755

Summary suggests that at the first level of the factor spd_des (but not the second level), data are significantly different from the grand mean (Intercept). On the other hand, anova is telling me that spd_des is non-significant (using alpha=0.05). Note that I am using type III sum of squares for anova, so I would have expected similar p-values as in Summary (although the statistical tests are different, t-test and F-test respectively). Maybe this is my own misconception, and I would like to understand this issue. What result should I use?

While that is my main concern, this issue raises another important and more general question that for me is still puzzling. What method should one use to perform models selection? Using the summary of the fit, one obtains different p-values for each level of the factors (in my case, I obtain p-values both for spd_des1 and for spd_des2), while using anova there is only one p-value for the factor independently of its levels. What should one do if the p-values associated to the different levels of a factors (output of summary) are inconsistent (i.e. some significant and others non-significant). Should one keep the factor or remove it from the model? (NOTE: I know this does not apply to the case I presented here because I cannot remove spd_des if there are still interactions with it, but I would like to understand this issue in general.) Thanks!

Answer 1

These results aren't inconsistent, as the tests are evaluating different hypotheses.

The F-test in ANOVA examines whether there are significant differences in response among all levels of the factor in question. The t-test for the individual coefficients, with sum contrasts as you have used, examines whether any individual response differs from the grand mean (that is, has a coefficient different from 0). The ANOVA for spd_des doesn't reach the usual p < 0.05 threshold for significance, but it's close enough that I wouldn't be surprised that one of its levels is associated with a value "significantly" different from from the grand mean in a simple t-test comparing its coefficient value against 0.

As to which result you should use, that depends on the purpose of your modeling.

If you are trying to convince someone of the importance of spd_des based solely on this model, then the classic ANOVA practice has been to examine the overall F-test for the factor first. If that is not significant, then one would not look for "significant" differences among individual levels of that factor. There is nothing preventing you from reporting both the F-test and the t-test results, but you should not argue that that spd_des is significantly related to response without a significant F-test. You could argue that more study is needed to evaluate the influence of spd_des.

But if this is the beginning of a larger study, or if your model is intended to be used for predictions, then you should be more forgiving of "insignificant" p-values. For a continuing course of study it would seem unwise to ignore in future work a predictor like spd_des; remember that avoiding Type I errors (false positives) entails a risk of Type II errors (false negatives). For predictive models it's generally better to err on the side of including more predictors than might be considered individually "significant." Any predictor that is reasonably expected to be related to response might help improve a predictive model, regardless of its individual "significance."