3
$\begingroup$

When we perform multiple linear regression with centered predictors (that is, $x_{ij}^c = x_{ij} - \bar{x}_j$) we get the same coefficients as with the original predictors but a different intercept. I've seen many conceptual explanations for this result but I can't find a rigorous derivation. The derivation is easy for simple linear regression but I haven't been able to work it out for multiple regression. I've tried the derivation in matrix notation and it goes off the rails pretty quickly. Can somebody show me the derivation?

$\endgroup$
2
$\begingroup$

If

$$y = \alpha + \beta_1 x_{i1} + \beta_2 x_{i2} + \epsilon$$

Then

\begin{align*} y_i &= \alpha + \beta_1 \bar x_1 - \beta_1\bar x_1 + \beta x_{i1} + \beta_2 x_{i2} + \epsilon_i \\ &= \underbrace{\tilde\alpha}_{\alpha + \beta_1 \bar x_1} + \beta_1\underbrace{\tilde x_{i1}}_{x_{i1} - \bar x_1} + \beta_2 x_{i2} + \epsilon_i \end{align*}

Clearly, we can do the same for $x_{i2}$ or any arbitrary number of covariates.

Now for the MLE. Suppose that the MLE exists and is unique. That is, it solves

$$ \hat{\vec{\gamma}} = \text{argmin}_{\vec{\gamma}} \lVert\vec{y}-(\vec{1},X)\vec{\gamma}\rVert^2, \qquad \vec{\gamma} = (\alpha, \vec{\beta}^\top)^\top$$

where $X$ the design matrix excluding the intercept and $\vec{1}$ is a vector of ones. Using the results from the The Partitioned Regression Model in this note then we have that

\begin{align} \hat{\vec{\beta}} &= (X^\top(I-P)X)^{-1}X^\top(I - P)\vec{y} \qquad P = \vec{1}^\top(\vec{1}^\top\vec{1})^{-1}\vec{1} \\ &=(X^\top C_nX)^{-1}X^\top C_n\vec{y} \end{align}

where $I$ is the identity matrix, $C_n = I - \frac{1}{n}\vec{1}\vec{1}^\top = I - P$ is the centering matrix, and we let $n$ denote the number of observations. Now, the centered design matrix excluding the intercept is

$$\tilde{X}=X - \vec{1}\bar{\vec{x}}^\top = C_n X$$

Thus, we clearly get the same $\vec{\beta}$. See also the Partitioned regression on the OLS wiki page. Surely, there is an easier way to argue that the two sets of slopes are equal.

Now that we know that the slopes are equal then the find that

\begin{align} \hat{\tilde \alpha} &= \text{argmin}_{\alpha} \lVert\vec{y}-(\vec{1},X)(\alpha,\hat{\vec{\beta}}^\top)^\top\rVert^2 \\ &= \lVert\vec{y} -\vec{1}\bar{\vec{x}}^\top\hat{\vec{\beta}} -(\vec{1},\tilde X)(\alpha,\hat{\vec{\beta}}^\top)^\top\rVert^2 \end{align}

which differs from the uncentered intercept by a $\bar{\vec{x}}^\top\hat{\vec{\beta}}$ term.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.