I'm reading Elements of Statistical Learning (2nd edition, 12th printing) and there are two things that are bothering me.

Regarding Equation 2.25, one is the training set Τ. I'm assuming that it is a function of two random variables X and Y, so when calculating the expected value, would the underlying probability density function be the joint distribution of X and Y?

Another thing is whether I should be interpreting y0-hat as a constant or a random variable? I know that it is an estimate of the test input point. If y0-hat is a random variable, what would be the underlying probability distribution?

I understand the derivation of decomposition of mean squared error into variance and bias, but when I look at Var(y0-hat), I can't help thinking that y0-hat is a constant and hence Var(y0-hat) = 0. What's wrong with my thinking?

Similar confusion arises for expected prediction error (equation 2.27), where the expected value is calculated with respect to y0|x0, which seems to me that y0 is a random variable and x0 is an event.

Sorry, I don't know how to format. Apologize if this was already posted. Any help would be greatly appreciated!

  • $\begingroup$ FYI, they use mathjax for math formatting. It’s pretty straight forward. You can find guides online quite easily if you’re interested. I’ll address your actual question below. $\endgroup$ Commented Jun 19 at 7:58
  • $\begingroup$ Oh also, add this to your post as not everyone is familiar with that text and seeing the problems helps. sas.upenn.edu/~fdiebold/NoHesitations/BookAdvanced.pdf $\endgroup$ Commented Jun 19 at 8:37

2 Answers 2


Yes, $X$ and $Y$ are random variables. Let me fill in some of the technical details.

For convenience, let $Z_i = (X_i,Y_i)$ be i.i.d. and define $\mathbf{Z} = \{Z_1,\ldots,Z_N\}$.

The estimator $\hat{y}$ is a function of all the observations in the training data (which is random) and $x$ (which is deterministic). In other words, $$\hat{y}= g(\mathbf{Z},x)$$

For example, the linear prediction fits this form $$ \hat{y} \equiv \hat{y}(x) = x \hat{\theta}_{OLS}$$

where $\hat{\theta}_{OLS}$ is the ordinary least squares estimate using all the observations in the training sample. In this case, $x$ is just a user-input whereas $\hat{\theta}$ needs to be estimated (with uncertainty). The true prediction is $f(x) = x \theta$.

This basic notation allows us to answer your question

  1. The expectation is taken over $\boldsymbol{Z}$, which includes all the $(X,Y)$ values in the training sample.
  2. The estimator $\hat{y}$ depends on the realizations of all the $Z_i$. You would get a different result if you randomly selected another training sample. Therefore, $Var(\hat{y}) > 0$.
  3. Yes, the MSE is a function of a deterministic input, $x$. However, remember that $\hat{y}$ is a function of both $\mathbf{Z}$ and $x$. The expectation integrates over $\mathbf{Z}$. There is no need to compute a conditional expectation.


  1. I would agree that the training set a function of X and Y and would be jointly distributed.
  2. $ \hat{y_0} $ is the expected value given $ x_0 $ and it is random. As for its distribution. It’s likely normally distributed if there’s a large training set. However, it isn’t always. You’d need to run diagnostic checks to see.
  3. $ \hat{y_0} $ is a random variable in this instance because supervised learning entails drawing repeated samples to create many training sets. Because expected values are a probability weighted average value based on the probability distribution of a given training set. As those sets are randomly selected, their distribution is also going to be random across iterations. The repeated measures using random samples every time injects variability. If you were to simply run the analysis once, then yes the expected value would be constant. But you aren’t so the variance is calculated using the data from the repeated sampling and testing.
  4. $ E_{y_0|x_0} $ refers to the expected value of $ y_0$ given $ x_0 $. Let’s say $ x_0 $ is a lower dosage of a medication. $ y_0 $ would represent the array of values reported by test subjects who received that lowest dose. They vary because people respond to medication differently. Just because the x-value is constant doesn’t mean that the outcome variable won’t be random. And technically x is random too, you’re just artificially restricting the range of values to evaluate the effects of the x variable at that level. So no, $ x_0 $ is not an event, it’s just a nonrandom subset of the x-variable data.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.