Mean Squared Error random variable

Question

I'm reading Elements of Statistical Learning (2nd edition, 12th printing) and there are two things that are bothering me.

Regarding Equation 2.25, one is the training set Τ. I'm assuming that it is a function of two random variables X and Y, so when calculating the expected value, would the underlying probability density function be the joint distribution of X and Y?

Another thing is whether I should be interpreting y0-hat as a constant or a random variable? I know that it is an estimate of the test input point. If y0-hat is a random variable, what would be the underlying probability distribution?

I understand the derivation of decomposition of mean squared error into variance and bias, but when I look at Var(y0-hat), I can't help thinking that y0-hat is a constant and hence Var(y0-hat) = 0. What's wrong with my thinking?

Similar confusion arises for expected prediction error (equation 2.27), where the expected value is calculated with respect to y0|x0, which seems to me that y0 is a random variable and x0 is an event.

Sorry, I don't know how to format. Apologize if this was already posted. Any help would be greatly appreciated!

Answer 1

Yes, $X$ and $Y$ are random variables. Let me fill in some of the technical details.

For convenience, let $Z_i = (X_i,Y_i)$ be i.i.d. and define $\mathbf{Z} = \{Z_1,\ldots,Z_N\}$.

The estimator $\hat{y}$ is a function of all the observations in the training data (which is random) and $x$ (which is deterministic). In other words, $$\hat{y}= g(\mathbf{Z},x)$$

For example, the linear prediction fits this form $$ \hat{y} \equiv \hat{y}(x) = x \hat{\theta}_{OLS}$$

where $\hat{\theta}_{OLS}$ is the ordinary least squares estimate using all the observations in the training sample. In this case, $x$ is just a user-input whereas $\hat{\theta}$ needs to be estimated (with uncertainty). The true prediction is $f(x) = x \theta$.

This basic notation allows us to answer your question

1. The expectation is taken over $\boldsymbol{Z}$, which includes all the $(X,Y)$ values in the training sample.
2. The estimator $\hat{y}$ depends on the realizations of all the $Z_i$. You would get a different result if you randomly selected another training sample. Therefore, $Var(\hat{y}) > 0$.
3. Yes, the MSE is a function of a deterministic input, $x$. However, remember that $\hat{y}$ is a function of both $\mathbf{Z}$ and $x$. The expectation integrates over $\mathbf{Z}$. There is no need to compute a conditional expectation.