Normally distributed $\epsilon$ for linear regression (ESL)

Question

While reading Elements of Statistical Learning, I encountered the following:

Let $$ Y = X^T \beta + \epsilon $$

with $\epsilon \sim \mathcal{N}(0, \sigma^2)$.

Fit an estimator to learn $\beta$ by least squares regression.

For an arbitrary point $x_0$ we have $$ \hat{y}_0 = x_0^T \beta + \sum_{i=1}^N \ell_i(x_0)\epsilon_i $$ where $\ell_i(x_0)$ is the $ith$ element of $$ \mathbf{X} \left( \mathbf{X}^T \mathbf{X} \right)^{-1} x_0 $$

I have read through the book from the beginning and am not sure where this comes from or how to interpret it. I have the basic mathematical background required to understand most of the material but am stumped here. Can someone help me understand

1. Where the summation term comes in, and
2. What exactly is $\mathbf{X} \left( \mathbf{X}^T \mathbf{X} \right)^{-1} x_0$ and how did it come about?

Answer 1

According to the model equation, $Y=X^T\beta+\epsilon$, $\beta$ should be of $p\times 1$, and $X$ should be $p\times n$, where $p$ is number of regressors including the constant if present, and $n$ is number of data samples. Therefore, $X(X^TX)^{-1}$ is of dimension $p\times n$, and the multiplication $X(X^TX)^{-1}x_0$ makes sense only when $x_0$ is of dimension $n$, which doesn't make sense since it is just one data sample and should have contained $p$ number of entries. I believe, there is typo present in the book, although I couldn't find it in its official errata.

What I'd suggest is to go for the common usage, $Y=X\beta+\epsilon$, where $X$ is $n\times p$. The book also uses this in Chapter 3. The LS solution for $\beta$ is $\hat{\beta}=(X^TX)^{-1}X^Ty$, and given a new data sample, $x_0$ (with dimension $p\times 1$), our estimate for the response will be $\hat{y_0}=x_0^T\hat{\beta}=x_0^T(X^TX)^{-1}X^Ty$. Substituting for $y$ leaves us with $$\hat{y_0}=x_0^T(X^TX)^{-1}X^T(X\beta+\epsilon)=x_0^T\beta+x_0^T(X^TX)^{-1}X^T\epsilon$$

The first term matches exactly with yours. The second term, as the first one, is just a number and it can be transposed, i.e. $$\hat{y_0}=x_0^T\beta+\epsilon^T\underbrace{[X(X^TX)^{-1}x_0]}_{l(x_0)}$$

Here, both $\epsilon$ and $l(x_0)$ are $n\times 1$ vectors, and this dot product can be written as sum of element-wise multiplication of corresponding indices: $$\epsilon^T[X(X^TX)^{-1}x_0]=\sum_{i=1}^n \epsilon_il_i(x_0)$$