0
$\begingroup$

I'm reading Andrew Ng's CS229 course notes on machine learning, and I'm at the part about Bias-Variance Tradeoff. Here, we're modeling our data as $y_i = f(x_i) + \epsilon_i$ where $\epsilon_i$ are i.i.d. with $\mathbb{E}[\epsilon_i]=0$ and $\text{Var}[\epsilon_i] = \sigma^2$. We learn a model $\hat{f}$ based off the training samples (and so this $\hat{f}(x)$ is actually a random variable for all $x$ in our test set).

Now, they go on to calculate the Test MSE as follows: $$ \begin{aligned} \text{Test MSE} & = \mathbb{E}\left((y-\hat{f}(x))^2\right) \\ & = \mathbb{E}\left((\epsilon + f(x) - \hat{f}(x))^2\right) \\ & = \mathbb{E}\left(\epsilon^2\right) + \mathbb{E}\left((f(x) - \hat{f}(x))^2\right) \\ & = \sigma^2 + \left(\mathbb{E}(f(x) - \hat{f}(x))\right)^2 + \text{Var}(f(x) - \hat{f}(x)) \\ & = \sigma^2 + \left(\text{Bias}(\hat{f})\right)^2 + \text{Var}(\hat{f}) \end{aligned} $$ I don't really have any problem with the formulas above, I can go from one to the next. What I don't understand is what the expectation is being taken over in the first place. What exactly does $\mathbb{E}((y - \hat{f}(x))^2)$ mean?

My intuition is telling me it's being taken over the training set $(x, y)\in\text{Test Set}$, with samples being drawn uniformly, but this doesn't make a lot of sense to me because I thought the whole point is that $\hat{f}$ is a random variable over the samples $\epsilon_i$, so shouldn't the expectation be over the $\epsilon_i$'s? If so, then are $x$ and $y$ just an arbitrary sample from the test set? How is $\text{Var}(\epsilon) = \sigma^2$, is $\epsilon$ our random variable here?

I'm just so confused about how exactly each of these expectations is being calculated. I get the gist of it, and I have no problem with formulas involving the expectation (like the fact that $\mathbb{E}[\epsilon^2] = \text{Var}(\epsilon) + \mathbb{E}[\epsilon]^2 = \text{Var}(\epsilon)$), I just don't understand what these expectations are being taken with respect to.

$\endgroup$
12
  • $\begingroup$ f(x) is the prediction of your model. If the model perfectly predicts y, then y-f(x) is zero, and that is the loss function being minimized here. When averaging this squared error over samples in the training/test set, you have a cost function called the mean squared error. Is all of this clear? $\endgroup$
    – hirschme
    Nov 29, 2018 at 1:32
  • $\begingroup$ @hirschme Yes I suppose, so you're saying the expectation is averaged over pairs $(x, y)$ in the training set? What exactly is $\epsilon$ then? $\endgroup$ Nov 29, 2018 at 1:34
  • $\begingroup$ Because you are modeling y to be equal to f(x) plus some additional noise $\epsilon$, then the best your model can perform is that error term, as it can't be predicted. $\endgroup$
    – hirschme
    Nov 29, 2018 at 1:35
  • $\begingroup$ @hirschme Yes, that makes sense, but that doesn't help me understand the precise meaning of the formulas above unfortunately. I can talk about Test MSE and the model learning in English confidently, but I can't really prove much about the formulas without knowing what they precisely correspond to (in English). $\endgroup$ Nov 29, 2018 at 1:39
  • $\begingroup$ If you now understand why we choose the term $\mathbb{E}((y - \hat{f}(x))^2)$ to describe the performance of a model $f(x)$ on data $(x,y)$ (regardles if it is training or test), then the rest is just algebra. In this case, the stochastic noise term $\epsilon$ is chosen to be standard gaussian noise, which has characteristics $\mathbb{E}(\epsilon) = 0$ and $Var(\epsilon) = \sigma^2$ ($\epsilon$ ~ $N(0, \sigma^2)$) $\endgroup$
    – hirschme
    Nov 29, 2018 at 1:46

1 Answer 1

2
$\begingroup$

In order to have a measure of the perfomance of our model $f(x)$ on approximating the map $x \rightarrow y$, we define an error (cost) function that measures the distance between our target $y$ and our estimate $\hat{y} = f(x)$ which we desire to minimize. An usual error term choice is the mean squared error MSE, which is the squared (direction insensitive) residuals, averaged over all sampled data pairs $(x_i,y_i), i=1,...,N$. This average is our approximation of the expected value of this residuals and can be written $\mathbb{E}[(y - f(x))^2]$.

The values of $f(x)$ will follow a distribution, governed by the inputs $x_i$, and other factors such as model choice, parameterization and so on. We can compare the performance of this model against a 'perfect' model $y = f(x) + \epsilon$. Having defined this perfect model $f(x)$, we call our model an estimate of this, thus now called $\hat{f}(x)$ and used in you equations.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.