1
$\begingroup$

The motivation of this question comes from the fact that sometimes you fit a regression model (let's say simple linear regression) and the coefficient of the explanatory variable is very significant, but then when you add another variable the first one becomes insignificant.

I understand that a lot of times this is due to multicollinearity which inflates the standard error thus making the t-statistic insignificant. Essentially because the two variables explain the same variation. My question is more about the claim I've read on multiple posts here that the t-test calculates the significance of a variable after accounting for what's explained by the other variables. When running a t-test manually I don't see where's the accounting for the other variables. How does that work?

Thanks.

$\endgroup$
5
  • $\begingroup$ What do you mean by running a t-test manually, $t=\frac{\bar{x}-\bar{y}}{s^2_{pooled}}$? $\endgroup$ – Dave Dec 4 '19 at 12:30
  • $\begingroup$ Meaning when you test the significance of the coefficient estimates by computing Variance-Covariance matrix, extracting standard errors and calculating the t-statistic $\endgroup$ – Metrician Dec 4 '19 at 12:38
  • $\begingroup$ How is that different from getting the t-stat for the regression coefficient? $\endgroup$ – Dave Dec 4 '19 at 12:40
  • $\begingroup$ I didn't say that it's different. Saying "manually" just means that when I go through the calculations step by step (as opposed to using a software) I don't see how the t-test accounts for what the other variables have explained. That's all. $\endgroup$ – Metrician Dec 4 '19 at 12:43
  • 1
    $\begingroup$ Let’s assume the usual nice properties we assume for linear regression parameter inference. The t-test of parameter $\beta_p$ is equivalent to F-testing the full model against the model with $\beta_p$ omitted. I will expand this into an answer later unless someone beats me to it, though I’d encourage you to do a simulation and try it for yourself before then. It is in this sense that it accounts for the variability caused by the group variable after the model has accounted for other sources of variability (the other variables in the regression). $\endgroup$ – Dave Dec 4 '19 at 12:50
1
$\begingroup$

Let’s assume the usual nice properties we assume for linear regression parameter inference. The t-test of parameter $\beta_p$ is equivalent to F-testing the full model against the model with $\beta_p$ omitted. Let's do a simulation where that $\beta_p$ corresponds to an indicator variable of group membership: control (0) versus treatment (1). The full model has one continuous variable, $X_1$, and then the binary group membership variable, $X_2$.

set.seed(2019)
N <- 1000
beta <- c(1,-0.2,0.01)
err <- rnorm(N,0,0.1)
x1 <- rnorm(N)
x2 <- rbinom(N,1,0.5)
X <- cbind(1,x1,x2)
y <- X %*% beta + err
L_full <- lm(y~x1+x2)
#
# Now let's build a reduced model that excludes x2
#
L_reduced <- lm(y~x1)
#
# Now let's find the F-stat for the full model against the reduced model
#
sse0 <- sum(resid(L_reduced)^2)
sse1 <- sum(resid(L_full)^2)
p0 <- dim(summary(L_reduced)$coefficients)[1]
p1 <- dim(summary(L_full)$coefficients)[1]
F <- ((sse0-sse1)/(p1-p0)) / ((sse1)/(N-p1))
#
# Print the F-stat
#
print(F)
#
# Print the squared t-stat for x2 in the full model
#
print((summary(L_full)$coefficients[3,3])^2)
#
# print the p-value from the F-test
# 
print(1-pf(F,p1-p0,N-p1))
#
# print the p-value from the t-test of x2 on the full model
#
print(summary(L_full)$coefficients[3,4])
#
# F = 10.40076
# t^2 = 10.40076
# p = 0.001300581 for the F-test on 1 and 997 degrees of freedom
# p = 0.001300581 for the regular printout from R's t-test of beta2 in the full model

As the last four lines show, the F-stat is equal to the squared t-stat, and, when the F-test has the appropriate degrees of freedom, both the F-test of full versus reduced and the t-test of $\beta_2$ give identical p-values.

The equation I use to calculate the F-stat comes from Agresti's Foundations of Linear and Generalized Linear Models on page 89:

$$ F= \dfrac{(SSE_{reduced}-SSE_{full})/(p_{full}-p_{reduced})}{SSE_{full}/(N-p_{full})} .$$

Agresti also gives the distribution under the null hypothesis that the reduced model holds (that is, the treatment is zero).

$$df_1 = p_{full}-p_{reduced}\\df_2 = N-p_{full}\\ F\sim F_{df_1,df_2} .$$

$p_{full}$ and $p_{reduced}$ are the number of parameters (including the intercept, if it is estimated) in the full and reduced models, respectively, and $N$ is the sample size.

Summarizing this simulation, we see that t-testing a group indicator variable is equivalent to F-testing the full model with that indicator variable against a reduced model that excludes the group variable. Therefore, the t-test on the group variable is accounting for variability caused by group membership after accounting for other sources of variability. Graphically, this can be visualized as parallel regression lines for each group, where the parameter on the group variable describes the distance (in the $y$ direction) between the groups after accounting for the other sources of variability.

Agresti, Alan. Foundations of Linear and Generalized Linear Models. Wiley, 2015.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.