4
$\begingroup$

I am currently reading "Elements of Statistical Learning II" and I am not quite sure about one thing in section 2.5 on p.24 and p.26

So at the end of p.24 they write the following:

Suppose that we know that the relationship between $Y$ and $X$ is linear,

$$Y=X^T\beta + \epsilon$$

where $\epsilon \sim N(0,\sigma^2)$ and we fit the model by least squares to the training data. For an arbitrary test point $x_0$, we have $\hat{y}_0=x_0^T\hat{\beta}$, which can be written as $\hat{y}_0=x_0^T\beta+\sum^N_{i=1}l_i(x_0)\epsilon_i$, where $l_i(x_0)$ is the $i$th elements of $X(X^TX)^{-1}x_0$.

I'm struggling to derive how did they arrive to this equation. I know that $\hat{\beta}=(X^TX)^{-1}X^TY$ and so $x_0^T\hat{\beta}=x_0^T(X^TX)^{-1}X^TY$

Since $Y=X^T\beta+\epsilon$ then $x_0^T\hat{\beta}=x_0^T(X^TX)^{-1}X^TY=x_0^T(X^TX)^{-1}X(X^T\beta+\epsilon)$

Well, i'm not sure how exactly to arrive to the equation they do. Can someone show me how?

$\endgroup$
1
  • $\begingroup$ Re "I know that:" change $\ldots XY$ to $\ldots X^\prime Y$ throughout so that the matrix product is defined, then consider the transpose of the coefficient matrix $x_0^\prime(X^\prime X)^{-}X^\prime$ and compare that to the book's statement. $\endgroup$
    – whuber
    Oct 5, 2021 at 18:46

2 Answers 2

2
$\begingroup$

The model formulation $\mathbf{Y} = \mathbf{X}^\text{T} \boldsymbol{\beta} + \boldsymbol{\varepsilon}$ is an unusual framing of the regression model (usually $\mathbf{X}$ would be the design matrix but here it is the transpose of the design matrix). With this formulation you have the OLS estimator $\boldsymbol{\hat{\beta}} = (\mathbf{X} \mathbf{X}^\text{T})^{-1} (\mathbf{X} \mathbf{Y})$. You have:

$$\begin{align} \hat{\mathbf{Y}} &= \mathbf{X}^\text{T} \boldsymbol{\hat{\beta}} \\[6pt] &= \mathbf{X}^\text{T} (\mathbf{X} \mathbf{X}^\text{T})^{-1} (\mathbf{X} \mathbf{Y}) \\[6pt] &= \mathbf{X}^\text{T} (\mathbf{X} \mathbf{X}^\text{T})^{-1} \mathbf{X}[\mathbf{X}^\text{T} \boldsymbol{\beta} + \boldsymbol{\varepsilon}] \\[6pt] &= \mathbf{X}^\text{T} (\mathbf{X} \mathbf{X}^\text{T})^{-1} \mathbf{X}\mathbf{X}^\text{T} \boldsymbol{\beta} + \mathbf{X}^\text{T} (\mathbf{X} \mathbf{X}^\text{T})^{-1} \mathbf{X} \boldsymbol{\varepsilon} \\[6pt] &= \mathbf{X}^\text{T} \boldsymbol{\beta} + [\mathbf{X}^\text{T} (\mathbf{X} \mathbf{X}^\text{T})^{-1} \mathbf{X}] \boldsymbol{\varepsilon} \\[6pt] &= \mathbf{X}^\text{T} \boldsymbol{\beta} + \mathbf{h} \boldsymbol{\varepsilon}, \\[6pt] \end{align}$$

where $\mathbf{h}$ is the hat matrix. So you have:

$$\begin{align} \hat{Y}_i = \mathbf{X}_i^\text{T} \boldsymbol{\beta} + [\mathbf{h} \boldsymbol{\varepsilon}]_i = \mathbf{X}_i^\text{T} \boldsymbol{\beta} + \sum_{j = 1}^n h_{i,j} \varepsilon_j. \\[6pt] \end{align}$$

where $h_{i,\ell} = [\mathbf{X}^\text{T} (\mathbf{X} \mathbf{X}^\text{T})^{-1} \mathbf{X}]_{i, \ell} = [\mathbf{X}^\text{T} (\mathbf{X} \mathbf{X}^\text{T})^{-1} \mathbf{X}_i]_{\ell} $. They use different notation than this (the correspondence is $\ell_j(\mathbf{X}_i) = [\mathbf{X}^\text{T} (\mathbf{X} \mathbf{X}^\text{T})^{-1} \mathbf{X}_i]_j$) but it is the same result.

$\endgroup$
2
$\begingroup$

Let's start with $\underset{N\times 1}{y}=\underset{N\times d}{X}.\underset{d\times 1}{\beta}+\underset{N\times 1}{\epsilon}$, note the dimension of the matrices / vectors underneath.

Hence,

$\sum\limits^N_{i=1}l_i(x_0)\epsilon_i$

$=\langle\;\underset{N\times d}{X}\underset{d \times d \quad}{(X^TX)^{-1}}\underset{d\times 1}{x_0}\quad,\quad \underset{N\times 1}{\epsilon}\;\rangle$, an inner product of two length-$N$ vectors

$=\left(X(X^TX)^{-1}x_0\right)^T.\epsilon$, since $\langle a, b \rangle = a^Tb$

$=\left(x_0^T(X^TX)^{-T}X^T\right).(y-X\beta)$, since $y=X\beta+\epsilon$

$=x_0^T(X^TX)^{-1}X^Ty-x_0^T(X^TX)^{-1}.(X^TX)\beta$, since $(X^TX)^T=X^TX$

$=x_0^T\hat{\beta}-x_0^T\beta$, where $\hat{\beta}=(X^TX)^{-1}X^Ty$

$=\hat{y}_0-x_0^T\beta$

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.