Demeaning variables in OLS

Question

I am analysing some data in R where I have information on $y$ and $x$. When I run $y = \alpha + \beta\cdot x$ I get the same coefficient on $x$ (i.e. $\beta$) as when I run $$y = \beta \cdot (x - \bar{x})$$ But when I run $$y - \bar{y} = \gamma \cdot x$$ I get a different coefficient on $x$.

I am really confused. Isn't the coefficient on $x$ meant to capture the covariance between $x$ and $y$, relative to the variance of $x$?

Why is it that demeaning the outcome variable ($y$) changes the coefficient on $x$ (when there is no intercept), but demeaning the independent variable ($x$) doesn't change it?

I tried to figure it out numerically but cannot recover this paradox.

Answer 1

The intercept-free model with centered $y$

But when I run $$y - \bar{y} = \gamma . x$$ I get a different coefficient on x.

If you rewrite that equation,

$$y = \bar{y} + \gamma x$$

then you see how it differs from your first case

$$y = \alpha + \beta x$$

---

The intercept-free model with centered $x$

The surprising part is not that $\beta$ and $\gamma$ are different, but that your second model, where you center the $x$ variable has the same slope with and without intercept.

$$y = \beta (x - \bar{x})$$

The coefficient $\beta$ is for simple linear regression $y_i = \alpha + \beta x_i + \epsilon$

$$\beta = \frac{\left(\frac{1}{n}\sum_{i=1}^n x_iy_i\right) - \bar{x}\bar{y} }{\left(\frac{1}{n}\sum_{i=1}^n x_i^2\right)-\bar{x}^2}$$

And for a model without intercept $y_i = \beta z_i + \epsilon$, with $z_i = x_i - \bar{x}$

$$\beta = \frac{\frac{1}{n}\sum_{i=1}^n z_iy_i}{\frac{1}{n}\sum_{i=1}^n z_i^2} = \frac{\left(\frac{1}{n}\sum_{i=1}^n x_iy_i\right) - \bar{x}\bar{y} }{\left(\frac{1}{n}\sum_{i=1}^n x_i^2\right)-\bar{x}^2} $$

where the second equation occurs when we fill in $z_i = x_i - \bar{x}$

---

Intuition

One way to see the least squares method is as a orthogonal projection of the observation $\mathbf{y}$ onto the space spanned by the identity vector $\mathbf{1}$ (the intercept) and the vector $\mathbf{x}$ or a matrix $\mathbf{X}$ containing those two vectors.

When the vectors that form that span are orthogonal then the projection is independent from whether or not a particular term is included. The different coefficients can be computed independently.

See also the question Intuition behind $(X^TX)^{-1}$ in closed form of w in Linear Regression . That term $(\mathbf{X^TX})^{-1}$ occurs in the computation of the projection and is a conversion of coefficients based on how much the different columns in $\mathbf{X}$ correlate with each other. It gives a conversion of coordinates $\boldsymbol{\alpha} = \mathbf{X^TY}$ to coordinates $\boldsymbol{\beta} = (\mathbf{X^T} \mathbf{X})^{-1}\mathbf{X^TY}$

When the columns in $\mathbf{X}$ are orthogonal, then the $(\mathbf{X^T} \mathbf{X})^{-1}$ has all the offdiagonal terms zero

$$\beta_i = (\mathbf{X^T} \mathbf{X})^{-1}_{ii} \mathbf{X^T_i} \mathbf{y} = \frac{\mathbf{X^T_i} \mathbf{y} }{\mathbf{X^T_i}\mathbf{X_i}}$$

Answer 2

It is not necessarily true that $\operatorname{cov}(x, y) / \operatorname{var}(x)$ is the slope in the case that there is no intercept.

The solution to the least squares problem for simple linear regression is easier to see if we write the formula for the slope equivalently as the following assuming an intercept term:

$$\hat{\beta} = \frac{\sum_i ((y_i - \bar{y})(x_i - \bar{x}))}{ \sum_i (x_i - \bar{x})^2}$$

but if there is no intercept the solution is

$$\hat{\beta} = \frac{\sum_i (y_ix_i) }{ \sum_i (x_i^2)}$$

Now the two are equal if $x_i - \bar{x}$ is used in place of $x_i$ in the latter as the denominators are then clearly the same and the numerators are equal because $\sum_i (-\bar{y}(x_i - \bar{x}))$ is zero but, in general, the two are not equal.

The place this often appears is when deriving $R^2$ since one then needs to use corresponding formulas for intercept and no-intercept cases.

Notice that if we use $\alpha + \beta(x_i - \bar{x})$ that the intercept and slope columns of the model matrix, i.e. a column of 1's and a column equal to $x - \bar{x}$, are orthogonal so the projection onto the latter is independent of the intercept; however, if we remove $-\bar{x}$ then they are no longer necessarily orthogonal.

Answer 3

I will discuss a general case of which your demeaning is a special case where $X_2$ is just the constant term, see Utility of the Frisch-Waugh theorem.

The Frisch Waugh Lovell theorem tells us that the coefficient $\hat\beta_1$ on the variables $X_1$ of a regression of $y$ on $X_1$ and $X_2$ can

(a) be obtained as usual by regressing $y$ on both $X_1$ and $X_2$ or

(b) by first regressing both, in separate regressions, $y$ and $X_1$ on $X_2$, saving the residuals of these regressions. These can, using the "residual maker matrix" $M_{X_2}=I-X_2(X_2'X_2)^{-1}X_2'$, be written as $M_{X_2}y$ and $M_{X_2}X_1$. Then, regress these residuals of the regression of $y$, $M_{X_2}y$, on those for the regressions for $X_1$, $M_{X_2}X_1$, or

(c) by only regressing $X_1$ on $X_2$, saving the residuals of these regressions and regress $y$ on the residuals for the regressions for $X_1$

Now, (b) can be written as, using the standard OLS formula, $$ \hat{\beta}_{\text{ols},1}=(X_1'M_{X_2}'M_{X_2}X_1)^{-1}X_1'M_{X_2}'M_{X_2}y $$ (to see this, just apply the general OLS expression $(X'X)^{-1}X'y$ to $X=M_{X_2}X_1$ and $y=M_{X_2}y$).

I skip that this is the same as (a), cf. e.g. Frisch-Waugh-Lovell theorem.

Why is the same as (c)?

Notice that $M_{X_2}$ is symmetric and idempotent (i.e., $M_{X_2}'M_{X_2}=M_{X_2}M_{X_2}=M_{X_2}$), so that we may as well write $$ \hat{\beta}_{\text{ols},1}=(X_1'M_{X_2}'M_{X_2}X_1)^{-1}X_1'M_{X_2}'y $$ (or $(X_1'M_{X_2}X_1)^{-1}X_1'M_{X_2}y$, if we wish), which is (c). This is what your regression of $y$ on demeaned $x$ amounts to.

It is (unless $M_{X_2}X_1=X_1\Leftrightarrow X_1'X_2=0$, i.e. unless regressors are orthogonal - in your case, unless $x$ has sample mean zero) however not the same as writing $$ (X_1'X_1)^{-1}X_1'M_{X_2}y. $$ This is what your regression of demeaned $y$ on $x$ amounts to.