# Linear regression formula with sum of residuals

The relationship in our model between X and Y is linear

$$Y = X^T\beta + \epsilon$$

For arbitrary test point $$x_0$$ we have prediction $$\hat{y_0} = x_0^{T}\hat{\beta}$$. Alternatively this can be written as $$\hat{y_0} = x_0^{T}\beta + \sum_{i=1}^Nl_i(x_0)\epsilon_i$$ where $$l_i(x_0)$$ is the ith element of $$X(X^TX)^{-1}x_0$$. In other words we can describe prediction using estimated $$\hat{\beta}$$ or assume ideal $$\beta$$ and add residuals.

How is the alternative formula derived?

• I'm not sure I follow your question. What are the estimates here? What are X and Y dimensions? Commented Dec 21, 2018 at 11:38

Consider

$$y_i = \mathbf x_i^\top \beta + \epsilon_i$$

for $$i=1,...,n$$ and with $$\beta \in \mathbb R^K$$ stack these to get

$$\mathbf y := \begin{bmatrix} y_1\\ \vdots \\ y_N \end{bmatrix} = \begin{bmatrix} \mathbf x_1^\top\\ \vdots \\\mathbf x_N^\top \end{bmatrix}\beta + \begin{bmatrix} \epsilon_1\\ \vdots \\ \epsilon_N \end{bmatrix} = \mathbf X \beta + \mathbf \epsilon$$

my $$\mathbf X$$ is corresponding to $$\mathbf X^\top$$ in the notation of the question however I prefer to change this to get notation consistent with what you usually find elsewhere. Then the OLS estimator is given as

$$\hat \beta_{OLS}:=(\mathbf X^\top \mathbf X)^{-1}\mathbf X^\top\mathbf y$$

and the predicted value is $$\hat{\mathbf y} := \mathbf X \hat \beta$$

dropping the subscript OLS for expediency. Insert into this the definition of the OLS estimator to get

$$\hat{\mathbf y} = \mathbf X (\mathbf X^\top \mathbf X)^{-1}\mathbf X^\top\mathbf y$$

Insert model equation $$\mathbf X \beta + \mathbf \epsilon$$ for $$\mathbf y$$ to get $$\hat{\mathbf y} = \mathbf X (\mathbf X^\top \mathbf X)^{-1}\mathbf X^\top\mathbf (\mathbf X \beta + \mathbf \epsilon) = \mathbf X\beta + \mathbf X (\mathbf X^\top \mathbf X)^{-1}\mathbf X^\top \mathbf \epsilon$$

pick out any row component $$j$$ to get

$$\hat y_j = \mathbf x_j^\top \beta + \{\mathbf X (\mathbf X^\top \mathbf X)^{-1}\mathbf X^\top \mathbf \epsilon\}_j$$

figure out how the $$j$$'th row component $$\{\mathbf X (\mathbf X^\top \mathbf X)^{-1}\mathbf X^\top \mathbf \epsilon\}_j$$ looks. Well consider this identity

$$\mathbf X (\mathbf X^\top \mathbf X)^{-1}\mathbf X^\top \mathbf \epsilon = \begin{bmatrix} \mathbf x_1^\top\\ \vdots \\\mathbf x_N^\top \end{bmatrix} (\mathbf X^\top \mathbf X)^{-1}\mathbf X^\top \mathbf \epsilon$$

from which it is apparent that the $$j$$'th row is

$$\mathbf x_j^\top (\mathbf X^\top \mathbf X)^{-1}\mathbf X^\top \mathbf \epsilon$$

hence I have

$$\hat y_j = \mathbf x_j^\top \beta + \mathbf x_j^\top (\mathbf X^\top \mathbf X)^{-1}\mathbf X^\top \mathbf \epsilon = \mathbf x_j^\top \beta +\sum_i \mathbf x_j^\top \mathbf z_i \mathbf \epsilon_i$$

where $$\mathbf z_i$$ is the $$i$$'th column of $$\mathbf Z:=(\mathbf X^\top \mathbf X)^{-1}\mathbf X^\top$$. Noting that $$\mathbf x_j^\top \mathbf z_i$$ is a linear combination it is informative to choose the letter $$l_i$$ to denote the linear combination $$l_i(\mathbf x_j) = \mathbf x_j^\top \mathbf z_i$$ in order to get

$$\hat y_j =\mathbf x_j^\top \beta +\sum_i l_i(\mathbf x_j) \mathbf \epsilon_i$$