A question about linear regression

Question

I have a simple question on linear regression or maybe this a linear algebra question.

For a simple linear model like:

$Y = \beta X + \varepsilon $

using OLS, we know that

$\beta = \frac{cov(X,Y)}{\sigma_x^2}$

or, in matrix notation,

$\beta = (X'Y)(X'X)^{-1}$

This formula implies that $cov(X,Y) = (X'Y)$.

Isn't this only true if the mean of $Y$ and $X$ is 0 ?

Answer 1

In short, yes.

The regression equation: $ y_i = \beta_1 x_i + \epsilon_i $ is omitting the constant term. This is assuming that the mean of y and x are zero.

In the case where we have: $$ y_i = \beta_0 + \beta_1 x_i + \epsilon_i $$ The estimate of $\beta_1$ can be obtained through: $$ \hat{\beta}_1 = \frac{cov(x_i y_i)}{\sigma_x^2} $$ OR $$ \hat{\beta} = (\textbf{X}'\textbf{X})^{-1}\textbf{X}'Y $$ EDIT: and $\beta_1$ is an element in the $\beta$ vector (the second according to the specification above).

X is BOTH the x terms AND a vector of ones.

Hope this helps your understanding.

Answer 2

As @AlanW noted, basically the answer is yes. The basic formula for linear regression is:

$$y_i = \beta_0 + \beta_1x_i + \epsilon_i$$

and you could translate it into (check here or here for more information):

$$y = m(y) + \frac{Cov(x,y)}{Var(x)}(x-m(x))$$

so if $m(y) = 0$ it becomes simply:

$$y = \frac{Cov(x,y)}{Var(x)}(x-m(x))$$

and if $m(x) = 0$ it becomes:

$$y = \frac{Cov(x,y)}{Var(x)}x$$

You can find more detailed description in many textbooks on regression, e.g. in "Regression Analysis by Example" by Samprit Chatterjee. However you have to remember that this is a simple case with one explanatory variable, if there are more explanatory variables it gets more complicated.

This is also a reason why centering (i.e. extracting the mean) makes life easier for estimating regression and is commonly used (especially with linear mixed models). Another real life problem that you can sometimes encounter is an error in statistical packages that says something like: Some predictor variables are on very different scales: consider rescaling (check here) and the root of the problem is that your variables are very different from each other and so it is hard to apply the formula. In this case rescaling them (extracting the mean and dividing by standard deviation) helps.

So the full answer is: the simple formula works well with similar, centered variables, however in real life you have to make correction to it by adding an intercept ($\beta_0$) to the equation and/or sometimes centering or scaling the variables, so it did not give you biased results.

Answer 3

For the linear model with a single regressor $y= \beta x + u$ it is not the case in general that $\hat \beta = \frac{\hat cov(X,Y)}{\hat \sigma_x^2}$, exactly because it is not clear whether $y$ and $x$ have mean zero. What is true is that

$$\hat \beta = \frac {\sum_{i=1}^ny_ix_i}{\sum_{i=1}^nx_i^2}$$

If we know that $y$ and $x$ have zero means (i.e. theoretical expected values, and irrespective of what happens in the sample means), then the above is the ratio of estimated covariance over variance (after multiply and divide by $(1/n)$ of course).

Assume now that we have a regression with multiple regressors including a constant term, $a$. Then partitioned regression results tell us that, in reality the $\beta$-vector containing the coefficients of the regressors (except the constant term) is estimated using the variables centered on sample means, i.e. in

$$y = a + \mathbf x'\beta + u$$

applied to a sample of size $n$, the OLS estimator for $\beta$ is

$$\hat \beta = (n^{-1}\mathbf {\tilde X'} \mathbf {\tilde X})^{-1} n^{-1}\mathbf {\tilde X}'\mathbf {\tilde y}$$

with $\mathbf {\tilde X} = \mathbf {X}-\mathbf {\bar X}$ and $\mathbf {\tilde y} = \mathbf {y}-\mathbf {\bar y}$ where the bar denotes sample means. Then $n^{-1} \mathbf {\tilde X}'\mathbf {\tilde y}$ is the vector containing the sample covariance between each regressor and the dependent variable, and $n^{-1}\mathbf {\tilde X'} \mathbf {\tilde X}$ is the sample variance-covariance matrix of the regressors (without including the contant term).

If we want to write $\beta$ to include the constant term (as is often the habit), then this relation is not immediately obvious (but it still holds).