9
$\begingroup$

The Bias Variance decomposition is a decomposition of an expectation, but I fail to follow what's actually assumed random specifically in this decomposition.

Take the specific regression example below in Eq. 7.9 from ESL. Is this expectation:

  1. across multiple fits of the model? (i.e. across multiple trainings?)
  2. across different data points x?

I get the impression that it's case #1, but that would be odd. The data is the data that we have for fitting, so what type of fits would they be?

E.g. is this looking at the generalization error? that is, for any dataset that we train with (not necessarily the one that we have) what $\text{Err}(x_0)$ we would see for a given generic point $x_0$? Or something else?

Improve this question
$\endgroup$
4
  • 1
    $\begingroup$ Expectation is over all possible samples. It’s very confusing here, but they somewhat mention that somewhere else IIRC $\endgroup$
    – user765195
    Commented May 30, 2020 at 5:16
  • $\begingroup$ Thanks @user765195 By all possible samples, what do you mean exactly? (samples of data points $x$? samples of datasets $X$?) The current answer I got says it's done to the prediction error on a fixed observation. Would you say that's not correct? $\endgroup$
    – Josh
    Commented May 30, 2020 at 12:04
  • 1
    $\begingroup$ Look at the discussion in Ch. 2. Over there, notice that the expectations are wrt the distribution of $\Tau$, the training set. $\endgroup$
    – user765195
    Commented May 31, 2020 at 8:30
  • 1
    $\begingroup$ The math in ESL is not very rigorous and I also had hard time to understand this paragraph. I found this post very helpful: stats.stackexchange.com/questions/164378/…, and also this course: work.caltech.edu/telecourse. $\endgroup$
    – SiXUlm
    Commented Jun 8, 2020 at 22:26

1 Answer 1

Reset to default
2
$\begingroup$

The Bias-Variance Decomposition is done to the prediction error on a fixed observation in the test set.

We assume we resample our training set again and again and re-train the model with each of the resampled train sets.

For example, the estimation of the error goes in this way: After we get $N$ train sets by resampling, we fit $N$ models with each of $N$ train sets. With the each of fitted models, we make a prediction on the same observation(OOS) in the test set. With the predictions, we will have $N$ predicted values, and the expected value of errors is calculated by taking the average of all the prediction errors.

The bias-variance decomposition states that the estimated error consists of error from bias, error from variance, and reducible error.

Improve this answer
$\endgroup$
4
  • $\begingroup$ Thanks Kevin. So what is are the expectations integrating over in the Equations I posted in my question? And why would it say $\text{Err}(x_0)$? (i.e. the expectations seem to be conditional on $X=x_0$)? $\endgroup$
    – Josh
    Commented Jun 2, 2020 at 12:19
  • $\begingroup$ @Josh It's conditional on $x_0$ because we want to test our model with the observation ($x_0$, $y_0$). So the error is calculated with the label $y_0$. The expectation is taken over the training sets. $\endgroup$
    – hbadger19042
    Commented Jun 2, 2020 at 13:32
  • $\begingroup$ You may check the Wikipedia article(en.wikipedia.org/wiki/Bias%E2%80%93variance_tradeoff). It explains more clearly. $\endgroup$
    – hbadger19042
    Commented Jun 2, 2020 at 13:38
  • $\begingroup$ Well if you fix the observation, it doesn't make sense to include the irreducible error. Remember $Y = f(X) + \epsilon$. If you fix the observation $(x_0, y_0)$ then $y_0 = f(x_0) + e_0$. Randomness is gone if you specify both $x_0$ and $y_0$. $\endgroup$
    – Antonios Sarikas
    Commented Feb 18, 2023 at 19:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.