1
$\begingroup$

For example, the space spanned by the columns in

$$\mathbf{X} = \begin{bmatrix} 0 & 0 \\ 1 & 0 \\ 0 & 1 \end{bmatrix}$$

is the y-z plane. Further,

$$\mathbf{X'X} = \begin{bmatrix} 1 & 0 \\ 0 & 1 \\ \end{bmatrix}$$

$$\mathbf{P = X(X'X)^{-1}X'} = \begin{bmatrix} 0 & 0 & 0 \\ 0 & 1 & 0 \\ 0 & 0 & 1 \end{bmatrix}$$

$$\mathbf{y} = \begin{bmatrix} 1 \\ 2 \\ 1 \end{bmatrix}$$

Then to project y onto the column space of X,

$$\mathbf{Py} = \begin{bmatrix} 0 \\ 2 \\ 1 \end{bmatrix}$$

I understand why this works. What I don't understand is how this works for linear regression - in the above example, each row represents a dimension in space I think (row 1 - x, row 2 - y, etc). In linear regression, the columns act as the dimensions. How does this transition work?

I apologize, because I know this is really stupid, but I still haven't pieced this together. Thank you for any help!

$\endgroup$

marked as duplicate by whuber regression May 25 at 19:47

This question has been asked before and already has an answer. If those answers do not fully address your question, please ask a new question.

0
$\begingroup$

Given OLS model: $y = X \beta + \epsilon$,

We wish to find the $\beta$ that minimizes the sum of squares error,

$L(\beta) = (y-X\beta)^T(y-X\beta)$

Differentiating $L$ with respect to $\beta$, setting the gradient to zero and solving for $\beta$:

$\nabla_{\beta} L(\beta) = 2X^T(y - X\beta) = 0$

We get that the OLS estimate of $\beta$ is: $\hat{\beta} = (X^TX)^{-1}X^T{y}$

The predicted response, $\hat{y} = X\hat{\beta} = X(X^TX)^{-1}X^T{y} = Py$ where $P = X(X^TX)^{-1}X^T$ is the projection matrix.

$\endgroup$

Not the answer you're looking for? Browse other questions tagged or ask your own question.