Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It only takes a minute to sign up.
Sign up to join this community
The relationship in our model between X and Y is linear
$Y = X^T\beta + \epsilon$
For arbitrary test point $x_0$ we have prediction $\hat{y_0} = x_0^{T}\hat{\beta}$. Alternatively this can be written as $\hat{y_0} = x_0^{T}\beta + \sum_{i=1}^Nl_i(x_0)\epsilon_i$ where $l_i(x_0)$ is the ith element of $X(X^TX)^{-1}x_0$. In other words we can describe prediction using estimated $\hat{\beta}$ or assume ideal $\beta$ and add residuals.
How is the alternative formula derived?
Consider
$$y_i = \mathbf x_i^\top \beta + \epsilon_i$$
for $i=1,...,n$ and with $\beta \in \mathbb R^K$ stack these to get
$$\mathbf y := \begin{bmatrix} y_1\\ \vdots \\ y_N \end{bmatrix} = \begin{bmatrix} \mathbf x_1^\top\\ \vdots \\\mathbf x_N^\top \end{bmatrix}\beta + \begin{bmatrix} \epsilon_1\\ \vdots \\ \epsilon_N \end{bmatrix} = \mathbf X \beta + \mathbf \epsilon $$
my $\mathbf X$ is corresponding to $\mathbf X^\top$ in the notation of the question however I prefer to change this to get notation consistent with what you usually find elsewhere. Then the OLS estimator is given as
$$\hat \beta_{OLS}:=(\mathbf X^\top \mathbf X)^{-1}\mathbf X^\top\mathbf y$$
and the predicted value is $$\hat{\mathbf y} := \mathbf X \hat \beta$$
dropping the subscript OLS for expediency. Insert into this the definition of the OLS estimator to get
$$\hat{\mathbf y} = \mathbf X (\mathbf X^\top \mathbf X)^{-1}\mathbf X^\top\mathbf y$$
Insert model equation $\mathbf X \beta + \mathbf \epsilon$ for $\mathbf y$ to get $$\hat{\mathbf y} = \mathbf X (\mathbf X^\top \mathbf X)^{-1}\mathbf X^\top\mathbf (\mathbf X \beta + \mathbf \epsilon) = \mathbf X\beta + \mathbf X (\mathbf X^\top \mathbf X)^{-1}\mathbf X^\top \mathbf \epsilon$$
pick out any row component $j$ to get
$$\hat y_j = \mathbf x_j^\top \beta + \{\mathbf X (\mathbf X^\top \mathbf X)^{-1}\mathbf X^\top \mathbf \epsilon\}_j$$
figure out how the $j$'th row component $\{\mathbf X (\mathbf X^\top \mathbf X)^{-1}\mathbf X^\top \mathbf \epsilon\}_j$ looks. Well consider this identity
$$\mathbf X (\mathbf X^\top \mathbf X)^{-1}\mathbf X^\top \mathbf \epsilon = \begin{bmatrix} \mathbf x_1^\top\\ \vdots \\\mathbf x_N^\top \end{bmatrix} (\mathbf X^\top \mathbf X)^{-1}\mathbf X^\top \mathbf \epsilon$$
from which it is apparent that the $j$'th row is
$$\mathbf x_j^\top (\mathbf X^\top \mathbf X)^{-1}\mathbf X^\top \mathbf \epsilon$$
hence I have
$$\hat y_j = \mathbf x_j^\top \beta + \mathbf x_j^\top (\mathbf X^\top \mathbf X)^{-1}\mathbf X^\top \mathbf \epsilon = \mathbf x_j^\top \beta +\sum_i \mathbf x_j^\top \mathbf z_i \mathbf \epsilon_i $$
where $\mathbf z_i$ is the $i$'th column of $\mathbf Z:=(\mathbf X^\top \mathbf X)^{-1}\mathbf X^\top$. Noting that $\mathbf x_j^\top \mathbf z_i$ is a linear combination it is informative to choose the letter $l_i$ to denote the linear combination $l_i(\mathbf x_j) = \mathbf x_j^\top \mathbf z_i$ in order to get
$$\hat y_j =\mathbf x_j^\top \beta +\sum_i l_i(\mathbf x_j) \mathbf \epsilon_i $$
