Intro to Statistical Learning - Solutions for 2.1 I am reading An Introduction to Statistical Learning with Applications in R (ISLR) and I wonder what would be the answer for exercise 2.1 part (d). The question is, If the variance of the error terms $$\sigma^2 = \mathrm{Var}(\epsilon)$$ is extremely high, a more flexible method would do worse or better? My intuition is it does not matter what method we choose, as it is an irreducible error, but most solutions I found said that the high variance of error terms means that the sample will have a lot of noise in the relationship. Therefore we should prefer an inflexible method that is less likely to over-fit to this noise.
Can someone explain it to me?
 A: It matters what you choose because a more flexible method may fit to the noise very easily, and you'll have to battle with it. As you mentioned, this is irreducible error, but an overfitted model will make much larger errors on the holdout set. Its aim is never reducing the irreducible error.
A: I would propose a thought experiment. Lets say you have some data generating process y=x + e with e a stochastic disturbance term with some distribution with a really high variance and x some number between 0 and 10. A very inflexible model would be y=constant=E[y], this would not even take x into account. You just 'guess' the mean of y every time.
Now imagine the opposite, you estimate an extremely flexible model y=f(X,B)+e. This model will fit every x to every y in your sample exactly. A perfect fit. Now we know the data generating process. We know every point y is mostly determined by your disturbance term e, since its value is probably much more extreme than the value of x. But your flexible model will attribute all variation in y to variation in X. Whereas in reality y is just very noisy without much structure. Attributing all variation to x is overfitting.
You ask "... a more flexible method would do worse or better?", I take it that by "better" you mean predicting y from new observations of x, given that y is generated by the same process. The inflexible method would predict E[y] whereas the flexible method would be all over the place.
