Variable is significant in multiple linear regression but not in t-test of the subgroups

Question

I am getting different p-values for a variable in t-test and its coefficient in multiple linear regression so I am unsure which one to believe. As an example, my hypothetical dataset consists of measurements of time taken to complete task A or B by cats and dogs, with 6 animals per group. I used R to analyse the data.

I want to know if there is a time difference according to the pet and task. So I do a 2-way ANOVA with interaction for pet and task.

# 2-way ANOVA with interaction
a1<-aov(time ~ pet * task, data)
summary(a1)

> summary(a1)
            Df Sum Sq Mean Sq F value   Pr(>F)    
pet          1 15.819  15.819  90.372 7.36e-09 ***
task         1  0.492   0.492   2.814    0.109    
pet:task     1  0.854   0.854   4.880    0.039 *  
Residuals   20  3.501   0.175                   

The ANOVA output tells me that time differs for pet (p-value significant) but not task (p-value not significant). The significant interaction term indicates that the time difference for pet depends on the type of task. Indeed the box plot shows that time looks different between cats doing task A vs task B, but not for dogs doing task A vs B.

To find the time difference between the subgroup cat.taskA vs the subgroup cat.taskB, I do a t-test and compare the results to a multiple linear regression. The t-test comparing cat.taskA vs cat.taskB shows that the p-value is not significant (p=0.05201). The mean difference is 0.6638174.

# t-test of task B vs A in cat only
dcat<-filter(data,pet=="cat")
t1<-t.test(time ~ task, dcat, var.equal=T)
t1

# standard error
t1$stderr

# mean difference
1.5931352-0.9293178

> t1

    Two Sample t-test

data:  time by task
t = 2.2048, df = 10, p-value = 0.05201
alternative hypothesis: true difference in means between group A and group B is not equal to 0
95 percent confidence interval:
 -0.007014563  1.334649298
sample estimates:
mean in group A mean in group B 
      1.5931352       0.9293178 

> # standard error
> t1$stderr
[1] 0.3010728

> # mean difference
> 1.5931352-0.9293178
[1] 0.6638174

When I fit a multiple linear regression model with interaction for pet and task, the coefficient for "taskB" is the time difference between cat.taskA and cat.taskB, which is the same as calculated manually (0.6638). However the coefficient is significant with a p-value of 0.0124 and the t-statistic is different from the t-test.

## multiple linear reg with interaction 
m1 <- lm(time ~ pet * task, data) 
summary(m1)

> summary(m1)

Call:
lm(formula = time ~ pet * task, data = data)

Residuals:
    Min      1Q  Median      3Q     Max 
-0.7877 -0.2352 -0.0554  0.2427  1.1500 

Coefficients:
             Estimate Std. Error t value Pr(>|t|)    
(Intercept)    1.5931     0.1708   9.327 1.01e-08 ***
petdog         1.2464     0.2415   5.160 4.76e-05 ***
taskB         -0.6638     0.2415  -2.748   0.0124 *  
petdog:taskB   0.7546     0.3416   2.209   0.0390 *  
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Residual standard error: 0.4184 on 20 degrees of freedom
Multiple R-squared:  0.8306,    Adjusted R-squared:  0.8052 
F-statistic: 32.69 on 3 and 20 DF,  p-value: 6.621e-08

I see that the t-statistic from the multiple linear regression is different from the t-test because of the smaller standard error for the "taskB" coefficient. But I am not understanding how this smaller standard error is made smaller in the multiple linear regression.

Should I believe the p-value from the t-test results or the multiple linear regression results? If I had only done a t-test comparing cat.taskA vs cat.taskB without bothering with ANOVA or multiple linear regression at all, I would have concluded that there is no difference in time. But the multiple linear regression is telling me there is a difference, unless it is not correct?

Answer 1

You wrote:

I am getting different p-values for a variable in t-test and its coefficient in multiple linear regression so I am unsure which one to believe.

Believe them all, as long as the assumptions of the model are met and you have programmed things correctly.

The ANOVA output tells me that time differs for pet (p-value significant) but not task (p-value not significant)

This is not correct. "Absence of evidence is not evidence of absence". Just because a result is not significant does not mean it isn't real.

To find the time difference between the subgroup cat.taskA vs the subgroup cat.taskB, I do a t-test and compare the results to a multiple linear regression. The t-test comparing cat.taskA vs cat.taskB shows that the p-value is not significant (p=0.05201). The mean difference is 0.6638174.

Doing the t-test is correct, but that doesn't let you control for other variables. Also, there is no need to compare to multiple regression. Also, "not significant" with p = 0.052 is sort of silly. Significance is not nearly as significant as people think it is. Look at the effect size (which you did) and its standard error.

I see that the t-statistic from the multiple linear regression is different from the t-test because of the smaller standard error for the "taskB" coefficient. But I am not understanding how this smaller standard error is made smaller in the multiple linear regression.

Because you are controlling for "pet" and for the interaction. Thus, in the multiple regression, the result for task is while holding pet constant and the effect for pet is while holding task constant.

Should I believe the p-value from the t-test results or the multiple linear regression results? If I had only done a t-test comparing cat.taskA vs cat.taskB without bothering with ANOVA or multiple linear regression at all, I would have concluded that there is no difference in time. But the multiple linear regression is telling me there is a difference, unless it is not correct?

Both are correct. But they are answers to different questions. One involves controlling for other variables, the other does not. Also, again, you should not conclude that there is no difference. That is not how p values work.

Answer 2

This question has multiple issues

Location of the intercept

One issue is that the main terms can be significant or not depending on the place of the intercept. This relates to R: Why does type III ANOVA require contrasts that sum to zero? and several other questions linked in that thread. Below is an example code that demonstrates the difference with a interaction term (which doesn't occur in the referenced question, but the principle is similar)

set.seed(1)
x1 = rbinom(100,1,0.5)
x2 = rbinom(100,1,0.5)
y = rnorm(100, x1+2*x2-0.3*x1*x2,1)

### change the coding 0 -> 1 and  1 -> 0
### this will effectively change the intercept
z1 = 1-x1 
z2 = 1-x2

summary(lm(y~1+x1+x2+x1*x2))

#lm(formula = y ~ 1 + x1 + x2 + x1 * x2)
#
#Coefficients:
#            Estimate Std. Error t value Pr(>|t|)    
#(Intercept)   0.2847     0.1989   1.431   0.1556    
#x1            0.4734     0.2813   1.683   0.0956 .  
#x2            1.6301     0.2663   6.121 2.03e-08 ***
#x1:x2         0.2200     0.3833   0.574   0.5674    



summary(lm(y~1+x1+z2+x1*z2))

#lm(formula = y ~ 1 + x1 + z2 + x1 * z2)
#
#Coefficients:
#            Estimate Std. Error t value Pr(>|t|)    
#(Intercept)   1.9148     0.1771  10.810  < 2e-16 ***
#x1            0.6934     0.2603   2.664  0.00907 ** 
#z2           -1.6301     0.2663  -6.121 2.03e-08 ***
#x1:z2        -0.2200     0.3833  -0.574  0.56736    

and see the difference between p-values 0.0956 and 0.00907 for the x1 coefficient.

The order of the variables

The analysis of variance can be done in different ways. An example question explaining this is: Order of variables in R's lm

So your anova results are not the same as the t-tests for the coefficient estimates in the multiple linear regression.

The single t-test versus the linear regression

In your t-test t1<-t.test(time ~ task, dcat, var.equal=T) you are comparing two groups which gives the same effect 0.6638174 as for the interaction term in linear model.

The reason for the different p-value is the difference in the estimate of the variance. The linear regression assumes that the variances are equal in all the groups. Because the dogs have a smaller variance, this influences your estimate for the variance with the cats.

Answer 3

Looking at the results and the boxplots, dogs have clearly larger values than cats. Looking at the difference between tasks A and B, values for task B look lower for cats but slightly larger for dogs (I'm not saying this is individually significant, see below). Taking this together, there is no evidence that values are systematically higher or lower overall between the two tasks (note that this is not what is tested by the test for taskB in model m1). There is evidence (although not that strong at $p=0.039$) that the difference between tasks plays out differently for cats and dogs.

Now you wonder how come that there is no significant difference between tasks looking at cats only, despite the data looking like there is a rather clear difference.

The first thing to acknowledge here is that the p-values of 0.039 and 0.052 are really not that different, and if we are not obsessed by significance thresholds, both can be interpreted as saying that there is some but rather weak indication that something is going on.

Note that the test for interaction in the aov output is based on all 24 observations, whereas the t-test is only based on 12 observations, 6 in each group. That isn't a very large number, and many things can happen when generating just 6 observations from two equal normal distributions (looking at cats only), and even a difference like the boxplot shows isn't totally unrealistic.

However, if there were no interaction, one wouldn't expect (after having looked at the cats and seeing that task B values are rather low) that the dogs would on average have higher values with rather low variance (even though not individually significant either). So the additional information how the dogs behave on task B compared to task A makes the test point out that there may well be an interaction going on. Of course the difference in p-values is low as said before, but still, adding this additional aspect from considering all 24 observations pushes the p-value below the 5% threshold here.

Answer 4

(a1 and m1 are the same model.)

What you're observing is that multiple regression is a more powerful approach than separate $t$-tests: By sharing a single error term, you end up with a larger number of degrees of freedom to compute the test statistic.

The $t$-test has no added value here.