I'm reading The Elements of Statistical Learning by T.Hastie I'm stuck on deriving EPE (formula 2.27) in 2.4 Statistical Decision Theory. Exercise 2.5(a)
In parallel I'm reading A Solution Manual and Notes for: The Elements of Statistical Learning by Jerome Friedman.. But it doesn't help in that case.

enter image description here enter image description here

Well I have two questions here but I will demonstrate what I did. Perhaps it helps to get my point.

$$EPE(x_{0}) = E_{y_{0}|x_{0}}E_{τ}(y_{0}-\hat{y}_{0}) = $$ $$E_{y_{0}|x_{0}}[E_{τ}(y^{2}_0)-(E_{τ}(y_0))^{2}+E_{τ}(\hat{y}^{2}_0)-(E_{τ}(\hat{y}_0))^{2}+(E_{τ}(\hat{y}_0)-y_0)^{2}|x_0] = $$ $$E_{y_{0}|x_{0}}[E_{τ}(y^{2}_0)|x_0]-E_{y_{0}|x_{0}}[(E_{τ}(y_0))^{2}|x_0]+E_{y_{0}|x_{0}}[E_{τ}(\hat{y}^{2}_0)|x_0]-E_{y_{0}|x_{0}}[(E_{τ}(\hat{y}_0))^{2}|x_0]+E_{y_{0}|x_{0}}[(E_{τ}(\hat{y}_0)-y_0)^{2}|x_0] = $$ $$E_{y_{0}|x_{0}}[y^{2}_0|x_0]-(E_{y_{0}|x_{0}}[y_0|x_0])^2+E_{τ}(\hat{y}^{2}_0)-(E_{τ}(\hat{y}_0))^{2}+(E_{τ}(\hat{y}_0)-E_{y_{0}|x_{0}}[y_0|x_0])^{2} = $$ $$Var(x_0|y_0) + Var(\hat{y}_0) - Bias^2(\hat{y}_0)$$

I'm done this so far. Probably with mistakes.

  1. I don't understand why Bias is 0. There is a little explanation In A Solution Manual and Notes book(page 9, Ex.2.5 Part(a)). The clue formula is in bottom of the image below. Why expected value of prediction is equal to $x^{T}_{0}$ multiplied by vector of errors? enter image description here

  2. How to derive $Var_{τ}(\hat{y}_0) = E_{τ}(x^{T}_{0}(X^{T}X)^{-1}x_{0}σ^{2}))$ from 2.27 formula?

Any hints or links will be appreciated


2 Answers 2


Equation 2.27 and Exercise 2.5 cover a lot of ground. It is easy to tell this, because Exercise 2.5 references Chapter 3, a future chapter. For example, Chapter 3 is where the $\beta$s come from in the proposed solution. It is where least squares is covered in detail. The wiki article on Ordinary Least Squares (OLS) uses the same notation and is pretty good.

I'm going to breakdown the problem in incredible detail because I think those answers are best and it is the way I think.

The expression of interest is, \begin{equation} EPE(x_{0})=E_{y_0|x_0}E_\tau \big( y_0 - \hat{y}_0 \big)^2. \tag{1}\label{problem2.27} \end{equation}

This is different from the MSE (Equation 2.25) because the problem defined in Equation 2.26 has a random variable ($\varepsilon$), so when a $x_0$ is selected a random number is pulled from the distribution of $\varepsilon$.

The problem is defined such that we have a training set $\tau$ that contains N examples each of $x_i, y_{i}$ and $\varepsilon_{i}$. The problem tells us that we know the relationship between these values is, \begin{equation} y_{i}=x_{i}^{T}\beta+\varepsilon_{i}, \end{equation} where $\beta$ is the true parameters that we want to estimate given the N data points in the training set $\tau$. OLS says that those estimated parameters are, \begin{equation} \hat\beta=\beta+(X^T X)^{-1}X^{T}\tilde\varepsilon, \end{equation} where $X$ is the design matrix (all the N training points $x_{i}$ turned into an N x p matrix), and $\tilde\varepsilon$ is a N x 1 vector of the actual errors in the training data. These are specific values that have been pulled from the distribution $\mathcal{N}(0,\sigma^2)$, they are normally distributed, but they are not random. This distinction is important because $E_{\tau}$ acts on $\tilde\varepsilon$ and $E_{y_{0}|x_{0}}$ acts on $\varepsilon$.

With the definitions out of the way, we can tackle Eq. \eqref{problem2.27} (2.27 from the book). As defined in the problem, we are given $x_0$, so we can estimate $\hat y_{0}=x_{0}^{T}\hat\beta=x_{0}^{T}\beta+x_{0}^{T}(X^T X)^{-1}X^{T}\tilde\varepsilon$, and we know from 2.26 that $y_{0}=x_{0}^{T}\beta+\varepsilon$. You can see right away that $y_{0}-\hat y_{0}$ does not have $\beta$ in it, they subtract out, so the fact that we don't know the true parameters $\beta$ doesn't matter. But we will take the longer route through Equation 2.25 to pull out the Bias and Variance separately.

Starting from the middle line in Equation 2.25, and noting that $f(x_{0})=y_{0}$, we can rewrite Eq. \eqref{problem2.27} as, \begin{equation} EPE(x_{0})=E_{y_0|x_0}\big[ E_{\tau}[\hat y_{0}-E_{\tau}(\hat y_{0})]^{2} + [E_{\tau}(\hat y_{0}) - y_{0}]^{2} \big], \tag{2}\label{inter2.27} \end{equation} where it is explicit that $E_{\tau}y_{0}=y_{0}$. The first term is the variance, so we turn to the second term and take the $E_{y_0|x_0}$ with us because it will not affect the variance.

\begin{equation} E_{y_0|x_0}[E_{\tau}(\hat y_{0}) - y_{0}]^{2} = E_{y_0|x_0}[E_{\tau}(\hat y_{0}) - x_{0}^{T}\beta - \varepsilon]^{2}=E_{y_0|x_0}[(E_{\tau}(\hat y_{0}) - x_{0}^{T}\beta)^2 + \varepsilon^{2} - 2\varepsilon(E_{\tau}(\hat y_{0}) - x_{0}^{T}\beta)], \end{equation} the first term here is the bias from Equation 2.25. It is not affected by $E_{y_0|x_0}$, while $E_{y_0|x_0}\varepsilon^2=\sigma^{2}$ and $E_{y_0|x_0}\varepsilon=0$ as defined by the distribution. Plugging this back into Eq. \eqref{inter2.27} we are left with,

\begin{equation} EPE(x_{0})=E_{\tau}[\hat y_{0}-E_{\tau}(\hat y_{0})]^{2} + (E_{\tau}(\hat y_{0}) - x_{0}^{T}\beta)^2 + \sigma^{2}, \end{equation} which is the variance, the bias and the 'additional variance' respectively.

Using $E_{\tau}\hat y_{0}=x_{0}^{T}\beta$ because $\tilde\varepsilon$ is pulled from $\mathcal{N}(0,\sigma^2)$ so $E_{\tau}\tilde\varepsilon=0$, it is immediately obvious the the bias is 0.

For your second question we have to look at the variance term, \begin{equation} Var_{\tau}(\hat y_{0})=E_{\tau}[\hat y_{0}-E_{\tau}(\hat y_{0})]^{2}=E_{\tau}[x_{0}^{T}(X^T X)^{-1}X^{T}\tilde\varepsilon]^{2}. \end{equation} It is important to note here that the term inside the brackets [] is a scalar, which can be verified by counting the dimensions of the matrices and vectors (as defined above) or noting that $\hat y_{0}$ is a scalar, so all terms in its definition must be too. Since they are scalars we can multiply them in any order, so we can write,

\begin{equation} E_{\tau}[x_{0}^{T}(X^T X)^{-1}X^{T}\tilde\varepsilon]^{2} = E_{\tau}[(x_{0}^{T}(X^T X)^{-1}X^{T}\tilde\varepsilon)(x_{0}^{T}(X^T X)^{-1}X^{T}\tilde\varepsilon)^{T}] = E_{\tau}[x_{0}^{T}(X^T X)^{-1}X^{T}\tilde\varepsilon\varepsilon^{T}X(X^T X)^{-1}x_{0}], \end{equation} where, as discussed before, $E_{\tau}$ only affects the $\varepsilon$, specifically $E_{\tau}\varepsilon\varepsilon^{T}=\sigma^{2}I_{N}$ and $I_{N}$ is the identity matrix of size N x N. Putting that together gives, \begin{equation} = E_{\tau}[x_{0}^{T}(X^T X)^{-1}X^{T}\tilde\varepsilon\tilde\varepsilon^{T}X(X^T X)^{-1}x_{0}] = x_{0}^{T}(X^T X)^{-1}X^{T}E_{\tau}[\tilde\varepsilon\tilde\varepsilon^{T}]X(X^T X)^{-1}x_{0} = x_{0}^{T}(X^T X)^{-1}(X^{T}X)(X^T X)^{-1}x_{0}\sigma^{2} = x_{0}^{T}(X^T X)^{-1}x_{0}\sigma^{2}. \end{equation}

Which is just about the result in 2.27. I'm not sure how $E_{\tau}x_{0}^{T}(X^T X)^{-1}x_{0}$ is different from $x_{0}^{T}(X^T X)^{-1}x_{0}$ because X is a design matrix that is deterministic. The solution you include from "A Solution and Notes" also seems to ignore this distinction. I don't think there is much to learn about the format of the variance in 2.27 beyond "if you write the result this way, you get this format".


I try to answer my questions. Looks like I realized how to derive the formula for the first question. But I'm halfway for the second one.

  1. $Bias^2(\hat{y}_{0}) = (E_{τ}(\hat{y}_0)-E_{y_{0}|x_{0}}[y_0|x_0])^{2} = (E_{τ}(\hat{y}_0)-x^{T}_{0}β)^{2} \text{- it is already known}$
    $E_{τ}(\hat{y}_0) = x^{T}_{0}E_{τ}(\hatβ) \text{ - to find expected y we need minimize error}$
    $x^{T}_{0}E_{τ}(\hatβ) = x^{T}_{0}β \text{ - because }\hatβ=β+(X^{T}X)^{-1}X^{T}ε = β \text{ and }E_{τ}(\hatβ)=β\text{ taking into account that } E_{τ}(ε)=0 \text{ by Gauss-Markov assumptions}$
    So $E_{τ}(\hat{y}_0) = x^{T}_{0}E_{τ}(\hatβ) = x^{T}_{0}β$ and $Bias^2(\hat{y}_{0}) = (x^{T}_{0}β-x^{T}_{0}β)^{2} = 0^2 = 0$

  2. Still stuck. I'm not sure that I'm doing this right but I've come to this so far.
    $Var_{τ}(\hat{y}_0) = Var_{τ}(x^{T}_{0}\hat{β}) = x^{T}_{0}Var_{τ}(\hat{β})=x^{T}_{0}(X^{T}X)^{-1}σ^{2}=???$


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.