18
$\begingroup$

I'm trying to work out how to perform some hypothesis testing on a linear regression (null hypothesis being no correlation). Every guide and page on the subject I run into seems to be using a t-test. But I don't understand what t-test for linear regression actually means. A t-test, unless I have a completely wrong understanding or mental model, is used to compare two populations. But the regressor and regressand are not samples of similar populations, and might not even be of the same unit, so it doesn't make sense to compare them.

So, when using a t-test on a linear regression, what is it that we're actually doing?

$\endgroup$
38
$\begingroup$

You're probably thinking of the two sample $t$ test because that's often the first place the $t$ distribution comes up. But really all a $t$ test means is that the reference distribution for the test statistic is a $t$ distribution. If $Z \sim \mathcal N(0,1)$ and $S^2 \sim \chi^2_d$ with $Z$ and $S^2$ independent, then $$ \frac{Z}{\sqrt{S^2 / d}} \sim t_d $$ by definition. I'm writing this out to emphasize that the $t$ distribution is just a name that was given to the distribution of this ratio because it comes up a lot, and anything of this form will have a $t$ distribution. For the two sample t test, this ratio appears because under the null the difference in means is a zero-mean Gaussian and the variance estimate for independent Gaussians is an independent $\chi^2$ (the independence can be shown via Basu's theorem which uses the fact that the standard variance estimate in a Gaussian sample is ancillary to the population mean, while the sample mean is complete and sufficient for that same quantity).

With linear regression we basically get the same thing. In vector form, $\hat \beta \sim \mathcal N(\beta, \sigma^2 (X^T X)^{-1})$. Let $S^2_j = (X^T X)^{-1}_{jj}$ and assume the predictors $X$ are non-random. If we knew $\sigma^2$ we'd have $$ \frac{\hat \beta_j - 0}{\sigma S_j} \sim \mathcal N(0, 1) $$ under the null $H_0 : \beta_j = 0$ so we'd actually have a Z test. But once we estimate $\sigma^2$ we end up with a $\chi^2$ random variable that, under our normality assumptions, turns out to be independent of our statistic $\hat \beta_j$ and then we get a $t$ distribution.


Here's the details of that: assume $y \sim \mathcal N(X\beta, \sigma^2 I)$. Letting $H = X(X^TX)^{-1}X^T$ be the hat matrix we have $$ \|e\|^2 = \|(I-H)y\|^2 = y^T(I-H)y. $$ $H$ is idempotent so we have the really nice result that $$ y^T(I-H)y / \sigma^2 \sim \mathcal \chi_{n-p}^2(\delta) $$ with non-centrality parameter $\delta = \beta^TX^T(I-H)X\beta = \beta^T(X^TX - X^T X)\beta = 0$, so actually this is a central $\chi^2$ with $n-p$ degrees of freedom (this is a special case of Cochran's theorem). I'm using $p$ to denote the number of columns of $X$, so if one column of $X$ gives the intercept then we'd have $p-1$ non-intercept predictors. Some authors use $p$ to be the number of non-intercept predictors so sometimes you might see something like $n-p-1$ in the degrees of freedom there, but it's all the same thing.

The result of this is that $E(e^Te / \sigma^2) = n-p$, so $\hat \sigma^2 := \frac{1}{n-p} e^T e$ works great as an estimator of $\sigma^2$.

This means that $$ \frac{\hat \beta_j}{\hat \sigma S_j}= \frac{\hat \beta_j}{S_j\sqrt{e^Te / (n-p)}} = \frac{\hat \beta_j}{\sigma S_j\sqrt{\frac{e^Te}{\sigma^2(n-p)}}} $$ is the ratio of a standard Gaussian to a chi squared divided by its degrees of freedom. To finish this, we need to show independence and we can use the following result:

Result: for $Z \sim \mathcal N_k(\mu, \Sigma)$ and matrices $A$ and $B$ in $\mathbb R^{l\times k}$ and $\mathbb R^{m\times k}$ respectively, $AZ$ and $BZ$ are independent if and only if $A\Sigma B^T = 0$ (this is exercise 58(b) in chapter 1 of Jun Shao's Mathematical Statistics).

We have $\hat \beta = (X^TX)^{-1}X^T y$ and $e = (I-H)y$ where $y \sim \mathcal N(X\beta, \sigma^2 I)$. This means $$ (X^TX)^{-1}X^T \cdot \sigma^2 I \cdot (I-H)^T = \sigma^2 \left((X^TX)^{-1}X^T - (X^TX)^{-1}X^TX(X^TX)^{-1}X^T\right) = 0 $$ so $\hat \beta \perp e$, and therefore $\hat \beta \perp e^T e$.

The upshot is we now know $$ \frac{\hat \beta_j}{\hat \sigma S_j} \sim t_{n-p} $$ as desired (under all of the above assumptions).


Here's the proof of that result. Let $C = {A \choose B}$ be the $(l+m)\times k$ matrix formed by stacking $A$ on top of $B$. Then $$ CZ = {AZ \choose BZ} \sim \mathcal N \left({A\mu \choose B\mu}, C\Sigma C^T \right) $$ where $$ C\Sigma C^T = {A \choose B} \Sigma \left(\begin{array}{cc} A^T & B^T \end{array}\right) = \left(\begin{array}{cc}A\Sigma A^T & A\Sigma B^T \\ B\Sigma A^T & B\Sigma B^T\end{array}\right). $$ $CZ$ is a multivariate Gaussian and it is a well-known result that two components of a multivariate Gaussian are independent if and only if they are uncorrelated, so the condition $A\Sigma B^T = 0$ turns out to be exactly equivalent to the components $AZ$ and $BZ$ in $CZ$ being uncorrelated.

$\square$

$\endgroup$
  • 3
    $\begingroup$ +1 always enjoy reading your answer. $\endgroup$ – Haitao Du May 2 '18 at 15:26
9
$\begingroup$

@Chaconne's answer is great. But here is a much shorter nonmathematical version!

Since the goal is to compute a P value, you first need to define a null hypothesis. Almost always, that is that the slope is actually horizontal so the numerical value for the slope (beta) is 0.0.

The slope fit from your data is not 0.0. Is that discrepancy due to random chance or due to the null hypothesis being wrong? You can't ever answer that for sure, but a P value is one way to sort-of-kind-of get at an answer.

The regression program reports a standard error of the slope. Compute the t ratio as the slope divided by its standard error. Actually, it is (slope minus null hypothesis slope) divided by the standard error, but the null hypothesis slope is nearly always zero.

Now you have a t ratio. The number of degrees of freedom (df) equals the number of data points minus the number of parameters fit by the regression (two for linear regression).

With those values (t and df) you can determine the P value with an online calculator or table.

It is essentially a one-sample t-test, comparing an observed computed value (the slope) with a hypothetical value (the null hypothesis).

$\endgroup$
  • 4
    $\begingroup$ The real question is why this is "essentially a one-sample t-test", and I don't see how it can become clear from your answer... $\endgroup$ – amoeba says Reinstate Monica May 8 '18 at 21:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.