5
$\begingroup$

Regarding a univariate OLS regression with a single categorical predictor (coded 0,1).

I am wrestling with the proof that $$t =\frac{b_1}{s(b_1)} $$

starting from the basic OLS estimator for the slope which is $$b_1= \frac{\sum (X_i - \bar{X})Y_i}{\sum(X_i - \bar{X})^2}. $$

I know that the first step is to show that the denominator $\sum(X_i - \bar{X})^2$ is equal to

$$\frac{n_1n_0}{n} $$

where $n_0$ and $n_1$ are the groups A and B where $X_i = 0$ and $1$ respectively.

I just can't get there, and know my partial summation algebra is lacking.

I am comfortable that $\sum(X_i - \bar{x})^2 = \sum(X_A - \bar{x})^2 + \sum(X_B - \bar{x})^2$ and then expanding each of these to the form $\sum X_i^2 - n\bar{x} ^2$ but can't get further than this:

$$\sum_{i=1}^{n_0} X_i^2 - n_0\bar{x} ^2 + \sum_{i=1}^{n_1} X_i^2 - n_1\bar{x} ^2.$$

I think the next step hinges on the fact that for the zero group $\sum X_i^2 = 0$ and for the 1 group $\sum X_i^2 = 1$.

Any advice on what I am missing to move forward here? Or if anyone can point me to a complete proof I'd be really grateful.

$\endgroup$
  • $\begingroup$ ok, I see, thanks, i thought it should be one sample t test. $\endgroup$ – Deep North Aug 9 '15 at 13:14
  • 1
    $\begingroup$ You are right about the sums by group. You would also need to figure out what $\bar x$ is in terms of $n_0$, $n_1$ and $n$. $\endgroup$ – StasK Aug 9 '15 at 14:15
  • $\begingroup$ I don't think that it directly helps, but still: In the last formula, the result is not 1, but rather $n_1$. And furthermore, in your formula for $b_1$, it should be $Y_i-\bar Y_i$ (or maybe you are assuming its average is 0). $\endgroup$ – Michael M Aug 10 '15 at 10:21
  • $\begingroup$ His formula for $b_1$ is correct when $X_i$ is binary, at the begginning, I also think it is wrong. Also I don't know what $X_A$ and $X_B$ stand for in his formula. $\endgroup$ – Deep North Aug 10 '15 at 11:57
2
$\begingroup$

This becomes easy when you reparameterize the problem.

Instead of using a slope and intercept, notice that when there are just two distinct values of the $x_i$ you can describe the fit by giving its value $\eta_0$ for $x=0$ and its value $\eta_1$ for $x=1$.

Figure

This example shows the data as red dots, the OLS fit as a dashed line, and summarizes the two groups with boxplots. Group $A$ is at the left and group $B$ at the right. The slope of the line is precisely the amount needed to go from the mean of group $A$, with $\eta_0$ near $10$, to the mean of group $B$, with $\eta_1$ near $13$.

Least squares requires you to choose values of these parameters that minimize the sum of squares of residuals. Since the value of $\eta_0$ affects the residuals only for group $A$ (where $x_i=0$) and $\eta_1$ affects the residuals only for group $B$ (where $x_i=1$), each will be estimated as the mean of its associated group. Because these means also happen to be the Maximum Likelihood estimates (as well as the OLS estimates), the ML estimate of the slope (which is also its OLS estimate) must be

$$b_1 = \frac{\hat\eta_1 - \hat\eta_0}{1-0} = \hat\eta_1 -\hat\eta_0,$$

which is just the difference in the group means. The OLS estimate of its variance (which does differ from the ML estimate, so we cannot exploit ML at this point) is the sum of squared residuals divided by the degrees of freedom, which is $n-2$. It should be equally obvious that this is precisely the pooled variance for the two-sample t-test. Consequently, $b_1/se(b_1)$ is exactly the same--and computed in exactly the same way--as the Student t statistic.

$\endgroup$
  • $\begingroup$ Thanks so much for this insight. It is one thing to solve algebraically but another to demonstrate in such an intuitive way... $\endgroup$ – Henry B Aug 15 '15 at 1:09
2
$\begingroup$

Thanks everyone for helping me out. Of course the first thing I needed to do was express $\bar{x}$ in terms of the ns. My final solution is below:

$$\sum_{i=1}^{n}(X_i - \bar{X})^2 = \sum_{i=1}^{(n_0 + n_1)} (X_i - \bar{X}) ^2 = \sum_{i=1}^{(n_0)} (X_i - \bar{X}) ^2 + \sum_{i=(n_0 + 1)}^{n} (X_i - \bar{X}) ^2 $$

$$ = \sum_{i=1}^{(n_0)} (0 - \frac{n_1}{n_0 + n_1}) ^2 + \sum_{i=(n_0 + 1)}^{n} (1 - \frac{n_1}{n_0 + n_1}) ^2 $$

$$ = \sum_{i=1}^{(n_0)} ( - \frac{n_1}{n_0 + n_1}) ^2 + n_1 (1 - \frac{n_1}{n_0 + n_1}) ^2 $$

$$ = n_0( - \frac{n_1}{n_0 + n_1}) ^2 + n_1 (1 - \frac{n_1}{n_0 + n_1}) ^2 $$

$$ = ( - \frac{n_0n_1}{n_0 + n_1}) ^2 + n_1 (1^2 - \frac{2n_1}{n_0 + n_1} + \frac{n_1^2}{(n_0 + n_1)^2} ) $$

$$ = \frac{n_0n_1^2}{(n_0 + n_1)^2} + \frac{n_1(n_0+n_1)^2}{(n_0 + n_1)^2} - \frac{2n_1^2 (n_0 + n_1)}{(n_0 + n_1)^2} + \frac{n_1^3}{(n_0 + n_1)^2} $$

$$ = \frac{n_0n_1^2 + n_1(n_0^2 + 2n_0n_1 + n_1^2) - 2n_1^2n_0 - 2n_1^3 + n_1^3}{n^2} $$

$$ = \frac{n_0n_1^2 + n_0^2n_1}{n^2} $$

$$ = \frac{n_0n_1(n_1 + n_0)}{n^2} $$

$$ = \frac{n_0n_1n}{n^2} $$

$$ = \frac{n_0n_1}{n}$$

$\endgroup$
0
$\begingroup$

If you just want to show that $\frac{b_1}{s(b_1)}$ has a $t$ distribution

Suppose your linear regression model is $ Y=Xb+\epsilon $ and your $\epsilon$ has a $N(0,\sigma^2)$ distribution. Therefore, $Y$ also has a $N(0,\sigma^2)$ distribution.

And solve the linear equation by matrix notation:

We can show that $b=(X'X)^{-1}X'Y$

And just note that $(X'X)^{-1}X'$ is a scalar vector, so $b$ has a normal distribution with $E(b)=0$ and variance is $(X'X)^{-1}X'\sigma^2((X'X)^{-1}X')'$

(If you just have one predictor and do not consider intercept, $(X'X)^{-1}X'$ is just a number.)

You will estimate the variance of $b$ by sample variance $s^2$

Then by Student theorem:

$\frac{b_1}{s(b1)}$ has a t distribution. Then you can use one sample t test to test whether $b_1=0$ i.e $T=\frac{b_1-0}{s(b_1)}$.

I think the key is to show $b$ is normal distributed through $Y$.

But I don't know how to prove by your method.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.