1
$\begingroup$

Here is what is what is written about the bias-variance tradeoff in "Introduction to Statistical Learning":

enter image description here

I am wondering, how to relate this to the linear regression model? Do we have in the bias-variance tradeoff that the $X$ vector used to construct $\hat{f}$ is itself random? So we need the distribution of $X$ as well? The reason this confuses be is that from what I recall in linear regression we only assume that $Y$ is random, but that the points $x_1,x_2,\ldots,x_n$ are fixed, and all the tests we make are under this assumption that $X$ is not random?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
0
$\begingroup$

You're right that in the basic linear regression model we assume that $X$ is not random. However, we do estimate $\beta$ and thus you can see that the fitted values (which are basically given by $\hat{f}$) are indeed random

$$\hat{y} = \hat{f}(X) = X \hat{\beta}$$.

See also here for an application to the linear regression model.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Thank you, yes I agree with what you are saying. But is it so that for the linear regression model we have that $Var(\hat{f}(x_0))$ and $[Bias(\hat{f}(x_0))]^2$ are calculated under the assumption that the when we have the training data $(x_1,y_1), \ldots (x_n,y_n)$ the points $x_k$ are fixed and the only randomness is in $y$? But in genereal the bias-variance tradeoff formula can also handle that the points $x_k$ in the training data are random?, which means that the randmness of $\hat{f}$ comes from randomness in $x_1,x_2,\ldots,x_n$ and $y_1,y_2,\ldots,y_n$? $\endgroup$
    – user394334
    Commented Oct 30 at 6:23
  • $\begingroup$ So the randomness is coming from $y_1,y_2,...,y_n$ as you've rightly point out. Are you referring to model selection? You can also have randomness in $\hat{f}$ by just specifying a 'wrong' model. Nevertheless, as far as I understand it, the $x_1,x_2,...,x_n$ itself are fixed and deterministic. $\endgroup$
    – Louki
    Commented Oct 30 at 7:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.