The relationship in our model between X and Y is linear

$Y = X^T\beta + \epsilon$

For arbitrary test point $x_0$ we have prediction $\hat{y_0} = x_0^{T}\hat{\beta}$. Alternatively this can be written as $\hat{y_0} = x_0^{T}\beta + \sum_{i=1}^Nl_i(x_0)\epsilon_i$ where $l_i(x_0)$ is the ith element of $X(X^TX)^{-1}x_0$. In other words we can describe prediction using estimated $\hat{\beta}$ or assume ideal $\beta$ and add residuals.

How is the alternative formula derived?

  • $\begingroup$ I'm not sure I follow your question. What are the estimates here? What are X and Y dimensions? $\endgroup$ Commented Dec 21, 2018 at 11:38

1 Answer 1



$$y_i = \mathbf x_i^\top \beta + \epsilon_i$$

for $i=1,...,n$ and with $\beta \in \mathbb R^K$ stack these to get

$$\mathbf y := \begin{bmatrix} y_1\\ \vdots \\ y_N \end{bmatrix} = \begin{bmatrix} \mathbf x_1^\top\\ \vdots \\\mathbf x_N^\top \end{bmatrix}\beta + \begin{bmatrix} \epsilon_1\\ \vdots \\ \epsilon_N \end{bmatrix} = \mathbf X \beta + \mathbf \epsilon $$

my $\mathbf X$ is corresponding to $\mathbf X^\top$ in the notation of the question however I prefer to change this to get notation consistent with what you usually find elsewhere. Then the OLS estimator is given as

$$\hat \beta_{OLS}:=(\mathbf X^\top \mathbf X)^{-1}\mathbf X^\top\mathbf y$$

and the predicted value is $$\hat{\mathbf y} := \mathbf X \hat \beta$$

dropping the subscript OLS for expediency. Insert into this the definition of the OLS estimator to get

$$\hat{\mathbf y} = \mathbf X (\mathbf X^\top \mathbf X)^{-1}\mathbf X^\top\mathbf y$$

Insert model equation $\mathbf X \beta + \mathbf \epsilon$ for $\mathbf y$ to get $$\hat{\mathbf y} = \mathbf X (\mathbf X^\top \mathbf X)^{-1}\mathbf X^\top\mathbf (\mathbf X \beta + \mathbf \epsilon) = \mathbf X\beta + \mathbf X (\mathbf X^\top \mathbf X)^{-1}\mathbf X^\top \mathbf \epsilon$$

pick out any row component $j$ to get

$$\hat y_j = \mathbf x_j^\top \beta + \{\mathbf X (\mathbf X^\top \mathbf X)^{-1}\mathbf X^\top \mathbf \epsilon\}_j$$

figure out how the $j$'th row component $\{\mathbf X (\mathbf X^\top \mathbf X)^{-1}\mathbf X^\top \mathbf \epsilon\}_j$ looks. Well consider this identity

$$\mathbf X (\mathbf X^\top \mathbf X)^{-1}\mathbf X^\top \mathbf \epsilon = \begin{bmatrix} \mathbf x_1^\top\\ \vdots \\\mathbf x_N^\top \end{bmatrix} (\mathbf X^\top \mathbf X)^{-1}\mathbf X^\top \mathbf \epsilon$$

from which it is apparent that the $j$'th row is

$$\mathbf x_j^\top (\mathbf X^\top \mathbf X)^{-1}\mathbf X^\top \mathbf \epsilon$$

hence I have

$$\hat y_j = \mathbf x_j^\top \beta + \mathbf x_j^\top (\mathbf X^\top \mathbf X)^{-1}\mathbf X^\top \mathbf \epsilon = \mathbf x_j^\top \beta +\sum_i \mathbf x_j^\top \mathbf z_i \mathbf \epsilon_i $$

where $\mathbf z_i$ is the $i$'th column of $\mathbf Z:=(\mathbf X^\top \mathbf X)^{-1}\mathbf X^\top$. Noting that $\mathbf x_j^\top \mathbf z_i$ is a linear combination it is informative to choose the letter $l_i$ to denote the linear combination $l_i(\mathbf x_j) = \mathbf x_j^\top \mathbf z_i$ in order to get

$$\hat y_j =\mathbf x_j^\top \beta +\sum_i l_i(\mathbf x_j) \mathbf \epsilon_i $$


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.