Proof of link between the OLS slope estimate and two sample t test statistic (categorical Xvar)

Question

Regarding a univariate OLS regression with a single categorical predictor (coded 0,1).

I am wrestling with the proof that $$t =\frac{b_1}{s(b_1)} $$

starting from the basic OLS estimator for the slope which is $$b_1= \frac{\sum (X_i - \bar{X})Y_i}{\sum(X_i - \bar{X})^2}. $$

I know that the first step is to show that the denominator $\sum(X_i - \bar{X})^2$ is equal to

$$\frac{n_1n_0}{n} $$

where $n_0$ and $n_1$ are the groups A and B where $X_i = 0$ and $1$ respectively.

I just can't get there, and know my partial summation algebra is lacking.

I am comfortable that $\sum(X_i - \bar{x})^2 = \sum(X_A - \bar{x})^2 + \sum(X_B - \bar{x})^2$ and then expanding each of these to the form $\sum X_i^2 - n\bar{x} ^2$ but can't get further than this:

$$\sum_{i=1}^{n_0} X_i^2 - n_0\bar{x} ^2 + \sum_{i=1}^{n_1} X_i^2 - n_1\bar{x} ^2.$$

I think the next step hinges on the fact that for the zero group $\sum X_i^2 = 0$ and for the 1 group $\sum X_i^2 = 1$.

Any advice on what I am missing to move forward here? Or if anyone can point me to a complete proof I'd be really grateful.

Answer 1

This becomes easy when you reparameterize the problem.

Instead of using a slope and intercept, notice that when there are just two distinct values of the $x_i$ you can describe the fit by giving its value $\eta_0$ for $x=0$ and its value $\eta_1$ for $x=1$.

This example shows the data as red dots, the OLS fit as a dashed line, and summarizes the two groups with boxplots. Group $A$ is at the left and group $B$ at the right. The slope of the line is precisely the amount needed to go from the mean of group $A$, with $\eta_0$ near $10$, to the mean of group $B$, with $\eta_1$ near $13$.

Least squares requires you to choose values of these parameters that minimize the sum of squares of residuals. Since the value of $\eta_0$ affects the residuals only for group $A$ (where $x_i=0$) and $\eta_1$ affects the residuals only for group $B$ (where $x_i=1$), each will be estimated as the mean of its associated group. Because these means also happen to be the Maximum Likelihood estimates (as well as the OLS estimates), the ML estimate of the slope (which is also its OLS estimate) must be

$$b_1 = \frac{\hat\eta_1 - \hat\eta_0}{1-0} = \hat\eta_1 -\hat\eta_0,$$

which is just the difference in the group means. The OLS estimate of its variance (which does differ from the ML estimate, so we cannot exploit ML at this point) is the sum of squared residuals divided by the degrees of freedom, which is $n-2$. It should be equally obvious that this is precisely the pooled variance for the two-sample t-test. Consequently, $b_1/se(b_1)$ is exactly the same--and computed in exactly the same way--as the Student t statistic.

Answer 2

Thanks everyone for helping me out. Of course the first thing I needed to do was express $\bar{x}$ in terms of the ns. My final solution is below:

$$\sum_{i=1}^{n}(X_i - \bar{X})^2 = \sum_{i=1}^{(n_0 + n_1)} (X_i - \bar{X}) ^2 = \sum_{i=1}^{(n_0)} (X_i - \bar{X}) ^2 + \sum_{i=(n_0 + 1)}^{n} (X_i - \bar{X}) ^2 $$

$$ = \sum_{i=1}^{(n_0)} (0 - \frac{n_1}{n_0 + n_1}) ^2 + \sum_{i=(n_0 + 1)}^{n} (1 - \frac{n_1}{n_0 + n_1}) ^2 $$

$$ = \sum_{i=1}^{(n_0)} ( - \frac{n_1}{n_0 + n_1}) ^2 + n_1 (1 - \frac{n_1}{n_0 + n_1}) ^2 $$

$$ = n_0( - \frac{n_1}{n_0 + n_1}) ^2 + n_1 (1 - \frac{n_1}{n_0 + n_1}) ^2 $$

$$ = ( - \frac{n_0n_1}{n_0 + n_1}) ^2 + n_1 (1^2 - \frac{2n_1}{n_0 + n_1} + \frac{n_1^2}{(n_0 + n_1)^2} ) $$

$$ = \frac{n_0n_1^2}{(n_0 + n_1)^2} + \frac{n_1(n_0+n_1)^2}{(n_0 + n_1)^2} - \frac{2n_1^2 (n_0 + n_1)}{(n_0 + n_1)^2} + \frac{n_1^3}{(n_0 + n_1)^2} $$

$$ = \frac{n_0n_1^2 + n_1(n_0^2 + 2n_0n_1 + n_1^2) - 2n_1^2n_0 - 2n_1^3 + n_1^3}{n^2} $$

$$ = \frac{n_0n_1^2 + n_0^2n_1}{n^2} $$

$$ = \frac{n_0n_1(n_1 + n_0)}{n^2} $$

$$ = \frac{n_0n_1n}{n^2} $$

$$ = \frac{n_0n_1}{n}$$

Answer 3

If you just want to show that $\frac{b_1}{s(b_1)}$ has a $t$ distribution

Suppose your linear regression model is $ Y=Xb+\epsilon $ and your $\epsilon$ has a $N(0,\sigma^2)$ distribution. Therefore, $Y$ also has a $N(0,\sigma^2)$ distribution.

And solve the linear equation by matrix notation:

We can show that $b=(X'X)^{-1}X'Y$

And just note that $(X'X)^{-1}X'$ is a scalar vector, so $b$ has a normal distribution with $E(b)=0$ and variance is $(X'X)^{-1}X'\sigma^2((X'X)^{-1}X')'$

(If you just have one predictor and do not consider intercept, $(X'X)^{-1}X'$ is just a number.)

You will estimate the variance of $b$ by sample variance $s^2$

Then by Student theorem:

$\frac{b_1}{s(b1)}$ has a t distribution. Then you can use one sample t test to test whether $b_1=0$ i.e $T=\frac{b_1-0}{s(b_1)}$.

I think the key is to show $b$ is normal distributed through $Y$.

But I don't know how to prove by your method.