Why do linear regression and ANOVA give different $p$-value in case of considering interaction between variable?

Question

I was trying to fit one time-series data (without replicates) using regression model. The data looks like follows:

> xx.2
          value time treat
    1  8.788269    1     0
    2  7.964719    6     0
    3  8.204051   12     0
    4  9.041368   24     0
    5  8.181555   48     0
    6  8.041419   96     0
    7  7.992336  144     0
    8  7.948658    1     1
    9  8.090211    6     1
    10 8.031459   12     1
    11 8.118308   24     1
    12 7.699051   48     1
    13 7.537120   96     1
    14 7.268570  144     1

Because of lack of replicates, I treat the time as continuous variable. Column "treat" shows the case and control data, respectively.

First, I fit the the model "value = time*treat" with "lm" in R:

summary(lm(value~time*treat,data=xx.2))

Call:
lm(formula = value ~ time * treat, data = xx.2)

Residuals:
     Min       1Q   Median       3Q      Max 
-0.50627 -0.12345  0.00296  0.04124  0.63785 

Coefficients:
             Estimate Std. Error t value Pr(>|t|)    
(Intercept)  8.493476   0.156345  54.325 1.08e-13 ***
time        -0.003748   0.002277  -1.646   0.1307    
treat       -0.411271   0.221106  -1.860   0.0925 .  
time:treat  -0.001938   0.003220  -0.602   0.5606    

The pvalue of time and treat is not significant.

While with anova, I got different results:

 summary(aov(value~time*treat,data=xx.2))
            Df Sum Sq Mean Sq F value Pr(>F)  
time         1 0.7726  0.7726   8.586 0.0150 *
treat        1 0.8852  0.8852   9.837 0.0106 *
time:treat   1 0.0326  0.0326   0.362 0.5606  
Residuals   10 0.8998  0.0900                 

The pvalue for time and treat changed.

With linear regression, if I am right, it means the time and treat has no significant influence on value, but with ANOVA, it means time and treat has significant influence on value.

Could someone explain to me why there is difference in these two methods, and which one to use?

Answer 1

The fit for lm() and aov() are identical but the reporting is different. The t tests are the marginal impact of the variables in question, given the presence of all the other variables. The F tests are sequential - so they test for the importance of time in the presence of nothing but the intercept, of treat in the presence of nothing but the intercept and time, and of the interaction in the presence of all the above.

Assuming you are interested in the significance of treat, I suggest you fit two models, one with, and one without, compare the two by putting both models in anova(), and use that F test. This will test treat and the interaction simultaneously.

Consider the following:

> xx.2 <- as.data.frame(matrix(c(8.788269, 1, 0,
+ 7.964719, 6, 0,
+ 8.204051, 12, 0,
+ 9.041368, 24, 0,
+ 8.181555, 48, 0,
+ 8.041419, 96, 0,
+ 7.992336, 144, 0,
+ 7.948658, 1, 1,
+ 8.090211, 6, 1,
+ 8.031459, 12, 1,
+ 8.118308, 24, 1,
+ 7.699051, 48, 1,
+ 7.537120, 96, 1,
+ 7.268570, 144, 1), byrow=T, ncol=3))
> names(xx.2) <- c("value", "time", "treat")
> 
> mod1 <- lm(value~time*treat, data=xx.2)
> anova(mod1)
Analysis of Variance Table

Response: value
           Df  Sum Sq Mean Sq F value  Pr(>F)  
time        1 0.77259 0.77259  8.5858 0.01504 *
treat       1 0.88520 0.88520  9.8372 0.01057 *
time:treat  1 0.03260 0.03260  0.3623 0.56064  
Residuals  10 0.89985 0.08998                  
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1 
> mod2 <- aov(value~time*treat, data=xx.2)
> anova(mod2)
Analysis of Variance Table

Response: value
           Df  Sum Sq Mean Sq F value  Pr(>F)  
time        1 0.77259 0.77259  8.5858 0.01504 *
treat       1 0.88520 0.88520  9.8372 0.01057 *
time:treat  1 0.03260 0.03260  0.3623 0.56064  
Residuals  10 0.89985 0.08998                  
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1 
> summary(mod2)
            Df Sum Sq Mean Sq F value Pr(>F)  
time         1 0.7726  0.7726   8.586 0.0150 *
treat        1 0.8852  0.8852   9.837 0.0106 *
time:treat   1 0.0326  0.0326   0.362 0.5606  
Residuals   10 0.8998  0.0900                 
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1 
> summary(mod1)

Call:
lm(formula = value ~ time * treat, data = xx.2)

Residuals:
     Min       1Q   Median       3Q      Max 
-0.50627 -0.12345  0.00296  0.04124  0.63785 

Coefficients:
             Estimate Std. Error t value Pr(>|t|)    
(Intercept)  8.493476   0.156345  54.325 1.08e-13 ***
time        -0.003748   0.002277  -1.646   0.1307    
treat       -0.411271   0.221106  -1.860   0.0925 .  
time:treat  -0.001938   0.003220  -0.602   0.5606    
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1 

Residual standard error: 0.3 on 10 degrees of freedom
Multiple R-squared: 0.6526,     Adjusted R-squared: 0.5484 
F-statistic: 6.262 on 3 and 10 DF,  p-value: 0.01154 

Answer 2

Peter Ellis' answer is excellent, but there is another point to be made. The $t$-test statistic (and its $p$-value) is a test of whether $\beta = 0$. The $F$-test on the anova() printout is whether the added variable significantly reduces the residual sum of squares.

The $t$-test is order-independent, while the $F$-test is not. Hence Peter's suggestion that you try the variables in different orders. It is also possible that variables significant in one test may not be significant in the other (and vice-versa).

My sense (and other contributors are welcome to correct me) is that when you're trying to predict phenomena (as in a systems application), you are most interested in reducing variance with the fewest predictors, and therefore want the anova() results. If you are trying to establish the marginal effect of $X$ on $y$, however, you will be most concerned with significance of your particular $\beta$ of interest, and all other variables will just control for alternate explanations your peer reviewers will try to find.