3
$\begingroup$

Why are the fitted values of a simple regression with intercept and dummy, estimated by OLS, just the group means belonging to the two groups of observations? I.e., why do we have that the fitted values for $x_i=t$ are given by

$$ \hat{y}=\bar{y}_{x_i=0} $$ when $x_i=0$ and $$ \hat{y}=\bar{y}_{x_i=1} $$ when $x_i=1$?

[Note: I ask and answer this question to fill in some missing pieces in this answer, but state it separately as it might be of independent interest, but has, to the best of my knowledge and search abilities, not been asked before.]

UPDATE: After the fact, I found an answer here. Since it uses somewhat different arguments and is maybe a little less explicit in the details, I still keep this one.

$\endgroup$
  • $\begingroup$ It is worth clarifying that this assumes parameters are estimated using OLS. It wouldn't be true if one were using LAD or some other robust regression. $\endgroup$ – The Laconic Jul 6 '18 at 14:36
  • $\begingroup$ true, I made an edit $\endgroup$ – Christoph Hanck Jul 6 '18 at 14:39
  • $\begingroup$ Any statistical procedure that minimizes a criterion which can be expressed as the sum of a function of one of the parameters and a function of the other parameter obviously produces the same answer as minimizing each function separately. Since simple regression minimizes the sum of squares of residuals, which splits over the sums by group, and its solution for each parameter separately is the group mean, the result follows. $\endgroup$ – whuber Jul 7 '18 at 20:44
2
$\begingroup$

The fitted values are given by $\bar{y}_{x_i=t}$ for those observations with $x_i=t$, $t=0,1$. That is, the specific group means:

It is well-known (see e.g. here) that the general expression for the slope coefficient in a simple linear regression is

$$ \hat{\beta}_1 = \frac{\sum_{i=1}^n(x_i-\bar{x})(y_i-\bar{y})}{\sum_{i=1}^n(x_i-\bar{x})^2} $$ Here, the regressor $x_i$ is a dummy that takes the values $\{0,1\}$. Let $n_t$ denote the number of observations such that $x_i=t$, so that $n=n_0+n_1$. Then, $\bar{x}=n_1/n$. By this rule, write the numerator of the estimator as $$ \sum_{i=1}^n(x_i-\bar{x})(y_i-\bar{y})=\sum_{i=1}^nx_iy_i-n\bar{y}\frac{n_1}{n}=n_1(\bar{y}_{x_i=1}-\bar{y}), $$ as multiplying by $x_i$ just "switches on" those $y_i$ for which $x_i=1$, and as the sum is the same as number of observations times average.

Similarly, we obtain for the numerator that $$ \sum_{i=1}^n(x_i-\bar{x})^2=\sum_{i=1}^nx_i^2-n\frac{n_1^2}{n^2}=n_1-\frac{n_1^2}{n}=\frac{n_0n_1}{n}, $$ where it was used that squaring $x_i$ of course still gives either zeros or ones and that $n=n_0+n_1$ gives $nn_1-n_1^2=n_0n_1$. Putting things together gives \begin{eqnarray*} \hat{\beta}_1 &=& \frac{nn_1(\bar{y}_{x_i=1}-\bar{y})}{n_0n_1}\\ &=& \frac{n(\bar{y}_{x_i=1}-\bar{y})}{n_0} \end{eqnarray*} where $\bar{y}_{x_i=t}$ denotes the sample mean of the $y_i$ belonging to the observations for which $x_i=t$. Now, write $$ \bar{y}=\frac{1}{n_0+n_1}(n_0\bar{y}_{x_i=0}+n_1\bar{y}_{x_i=1}) $$ Hence, \begin{eqnarray*} \hat{\beta}_1 &=& \frac{(n_0+n_1)(\bar{y}_{x_i=1}-\frac{1}{n_0+n_1}(n_0\bar{y}_{x_i=0}+n_1\bar{y}_{x_i=1}))}{n_0}\\ &=& \frac{(n_0+n_1)\bar{y}_{x_i=1}-n_0\bar{y}_{x_i=0}-n_1\bar{y}_{x_i=1}}{n_0}\\ &=& \frac{n_0\bar{y}_{x_i=1}-n_0\bar{y}_{x_i=0}}{n_0}\\ &=& \bar{y}_{x_i=1}-\bar{y}_{x_i=0} \end{eqnarray*} The intercept is, in general, given by $$ \hat\beta_0=\bar{y}-\hat{\beta}_1\bar{x} $$ Thus, writing $\sum_iy_i$ as the sum of group averages times their respective size, \begin{eqnarray*} \hat\beta_0&=&\frac{1}{n}(n_0\bar{y}_{x_i=0}+n_1\bar{y}_{x_i=1})-(\bar{y}_{x_i=1}-\bar{y}_{x_i=0})\frac{n_1}{n}\\ &=&\frac{n_0+n_1}{n}\bar{y}_{x_i=0}=\bar{y}_{x_i=0} \end{eqnarray*} Thus, the fitted values for $x_i=t$ are given by $$\hat{y}=\hat\beta_0+\hat\beta_1\cdot t=\bar{y}_{x_i=0}+(\bar{y}_{x_i=1}-\bar{y}_{x_i=0})t, $$ i.e. $$ \hat{y}=\bar{y}_{x_i=0} $$ when $x_i=0$ and $$ \hat{y}=\bar{y}_{x_i=1} $$ when $x_i=1$.

| cite | improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.