# Making the mindset transition - projection in OLS regression [duplicate]

For example, the space spanned by the columns in

$$\mathbf{X} = \begin{bmatrix} 0 & 0 \\ 1 & 0 \\ 0 & 1 \end{bmatrix}$$

is the y-z plane. Further,

$$\mathbf{X'X} = \begin{bmatrix} 1 & 0 \\ 0 & 1 \\ \end{bmatrix}$$

$$\mathbf{P = X(X'X)^{-1}X'} = \begin{bmatrix} 0 & 0 & 0 \\ 0 & 1 & 0 \\ 0 & 0 & 1 \end{bmatrix}$$

$$\mathbf{y} = \begin{bmatrix} 1 \\ 2 \\ 1 \end{bmatrix}$$

Then to project y onto the column space of X,

$$\mathbf{Py} = \begin{bmatrix} 0 \\ 2 \\ 1 \end{bmatrix}$$

I understand why this works. What I don't understand is how this works for linear regression - in the above example, each row represents a dimension in space I think (row 1 - x, row 2 - y, etc). In linear regression, the columns act as the dimensions. How does this transition work?

I apologize, because I know this is really stupid, but I still haven't pieced this together. Thank you for any help!

## marked as duplicate by whuber♦ regression StackExchange.ready(function() { if (StackExchange.options.isMobile) return; $('.dupe-hammer-message-hover:not(.hover-bound)').each(function() { var$hover = $(this).addClass('hover-bound'),$msg = $hover.siblings('.dupe-hammer-message');$hover.hover( function() { $hover.showInfoMessage('', { messageElement:$msg.clone().show(), transient: false, position: { my: 'bottom left', at: 'top center', offsetTop: -7 }, dismissable: false, relativeToBody: true }); }, function() { StackExchange.helpers.removeMessages(); } ); }); }); May 25 at 19:47

• – whuber May 25 at 19:46

Given OLS model: $$y = X \beta + \epsilon$$,

We wish to find the $$\beta$$ that minimizes the sum of squares error,

$$L(\beta) = (y-X\beta)^T(y-X\beta)$$

Differentiating $$L$$ with respect to $$\beta$$, setting the gradient to zero and solving for $$\beta$$:

$$\nabla_{\beta} L(\beta) = 2X^T(y - X\beta) = 0$$

We get that the OLS estimate of $$\beta$$ is: $$\hat{\beta} = (X^TX)^{-1}X^T{y}$$

The predicted response, $$\hat{y} = X\hat{\beta} = X(X^TX)^{-1}X^T{y} = Py$$ where $$P = X(X^TX)^{-1}X^T$$ is the projection matrix.