# Making the mindset transition - projection in OLS regression [duplicate]

For example, the space spanned by the columns in

$$\mathbf{X} = \begin{bmatrix} 0 & 0 \\ 1 & 0 \\ 0 & 1 \end{bmatrix}$$

is the y-z plane. Further,

$$\mathbf{X'X} = \begin{bmatrix} 1 & 0 \\ 0 & 1 \\ \end{bmatrix}$$

$$\mathbf{P = X(X'X)^{-1}X'} = \begin{bmatrix} 0 & 0 & 0 \\ 0 & 1 & 0 \\ 0 & 0 & 1 \end{bmatrix}$$

$$\mathbf{y} = \begin{bmatrix} 1 \\ 2 \\ 1 \end{bmatrix}$$

Then to project y onto the column space of X,

$$\mathbf{Py} = \begin{bmatrix} 0 \\ 2 \\ 1 \end{bmatrix}$$

I understand why this works. What I don't understand is how this works for linear regression - in the above example, each row represents a dimension in space I think (row 1 - x, row 2 - y, etc). In linear regression, the columns act as the dimensions. How does this transition work?

Thank you for any help!

• – whuber May 25 '19 at 19:46

Given OLS model: $$y = X \beta + \epsilon$$,

We wish to find the $$\beta$$ that minimizes the sum of squares error,

$$L(\beta) = (y-X\beta)^T(y-X\beta)$$

Differentiating $$L$$ with respect to $$\beta$$, setting the gradient to zero and solving for $$\beta$$:

$$\nabla_{\beta} L(\beta) = 2X^T(y - X\beta) = 0$$

We get that the OLS estimate of $$\beta$$ is: $$\hat{\beta} = (X^TX)^{-1}X^T{y}$$

The predicted response, $$\hat{y} = X\hat{\beta} = X(X^TX)^{-1}X^T{y} = Py$$ where $$P = X(X^TX)^{-1}X^T$$ is the projection matrix.