# How is Test MSE being calculated here?

I'm reading Andrew Ng's CS229 course notes on machine learning, and I'm at the part about Bias-Variance Tradeoff. Here, we're modeling our data as $$y_i = f(x_i) + \epsilon_i$$ where $$\epsilon_i$$ are i.i.d. with $$\mathbb{E}[\epsilon_i]=0$$ and $$\text{Var}[\epsilon_i] = \sigma^2$$. We learn a model $$\hat{f}$$ based off the training samples (and so this $$\hat{f}(x)$$ is actually a random variable for all $$x$$ in our test set).

Now, they go on to calculate the Test MSE as follows: \begin{aligned} \text{Test MSE} & = \mathbb{E}\left((y-\hat{f}(x))^2\right) \\ & = \mathbb{E}\left((\epsilon + f(x) - \hat{f}(x))^2\right) \\ & = \mathbb{E}\left(\epsilon^2\right) + \mathbb{E}\left((f(x) - \hat{f}(x))^2\right) \\ & = \sigma^2 + \left(\mathbb{E}(f(x) - \hat{f}(x))\right)^2 + \text{Var}(f(x) - \hat{f}(x)) \\ & = \sigma^2 + \left(\text{Bias}(\hat{f})\right)^2 + \text{Var}(\hat{f}) \end{aligned} I don't really have any problem with the formulas above, I can go from one to the next. What I don't understand is what the expectation is being taken over in the first place. What exactly does $$\mathbb{E}((y - \hat{f}(x))^2)$$ mean?

My intuition is telling me it's being taken over the training set $$(x, y)\in\text{Test Set}$$, with samples being drawn uniformly, but this doesn't make a lot of sense to me because I thought the whole point is that $$\hat{f}$$ is a random variable over the samples $$\epsilon_i$$, so shouldn't the expectation be over the $$\epsilon_i$$'s? If so, then are $$x$$ and $$y$$ just an arbitrary sample from the test set? How is $$\text{Var}(\epsilon) = \sigma^2$$, is $$\epsilon$$ our random variable here?

I'm just so confused about how exactly each of these expectations is being calculated. I get the gist of it, and I have no problem with formulas involving the expectation (like the fact that $$\mathbb{E}[\epsilon^2] = \text{Var}(\epsilon) + \mathbb{E}[\epsilon]^2 = \text{Var}(\epsilon)$$), I just don't understand what these expectations are being taken with respect to.

• f(x) is the prediction of your model. If the model perfectly predicts y, then y-f(x) is zero, and that is the loss function being minimized here. When averaging this squared error over samples in the training/test set, you have a cost function called the mean squared error. Is all of this clear? Nov 29, 2018 at 1:32
• @hirschme Yes I suppose, so you're saying the expectation is averaged over pairs $(x, y)$ in the training set? What exactly is $\epsilon$ then? Nov 29, 2018 at 1:34
• Because you are modeling y to be equal to f(x) plus some additional noise $\epsilon$, then the best your model can perform is that error term, as it can't be predicted. Nov 29, 2018 at 1:35
• @hirschme Yes, that makes sense, but that doesn't help me understand the precise meaning of the formulas above unfortunately. I can talk about Test MSE and the model learning in English confidently, but I can't really prove much about the formulas without knowing what they precisely correspond to (in English). Nov 29, 2018 at 1:39
• If you now understand why we choose the term $\mathbb{E}((y - \hat{f}(x))^2)$ to describe the performance of a model $f(x)$ on data $(x,y)$ (regardles if it is training or test), then the rest is just algebra. In this case, the stochastic noise term $\epsilon$ is chosen to be standard gaussian noise, which has characteristics $\mathbb{E}(\epsilon) = 0$ and $Var(\epsilon) = \sigma^2$ ($\epsilon$ ~ $N(0, \sigma^2)$) Nov 29, 2018 at 1:46

In order to have a measure of the perfomance of our model $$f(x)$$ on approximating the map $$x \rightarrow y$$, we define an error (cost) function that measures the distance between our target $$y$$ and our estimate $$\hat{y} = f(x)$$ which we desire to minimize. An usual error term choice is the mean squared error MSE, which is the squared (direction insensitive) residuals, averaged over all sampled data pairs $$(x_i,y_i), i=1,...,N$$. This average is our approximation of the expected value of this residuals and can be written $$\mathbb{E}[(y - f(x))^2]$$.
The values of $$f(x)$$ will follow a distribution, governed by the inputs $$x_i$$, and other factors such as model choice, parameterization and so on. We can compare the performance of this model against a 'perfect' model $$y = f(x) + \epsilon$$. Having defined this perfect model $$f(x)$$, we call our model an estimate of this, thus now called $$\hat{f}(x)$$ and used in you equations.