0
$\begingroup$

I am just grasping the bias variance trade-off as it is explained by the MSE heuristic. We have that if $y = f(x) + \epsilon$ for $\epsilon \sim N(0,\sigma^2)$ we can show that

\begin{aligned} MSE(y, \hat y) &= Var(\epsilon) + Bias^2(\hat y) + Var(\hat y) \\ &= Var(\epsilon) + Var(\hat y) \qquad \text{ because we assume OLS is unbiased} \end{aligned}

I am confused because when we solve linear regression via MLE, we assume the same $y = f(x) + \epsilon$ for $\epsilon$ normally distributed with variance

$$Var(\epsilon) = MSE(y, \hat y)$$

What happened to the $Var(\hat y)$ term?

Thank you!

$\endgroup$

1 Answer 1

0
$\begingroup$

Hi: In your second formula, the bias is assumed to be zero so there is no bias term. The variance term is the MSE

$\endgroup$
2
  • $\begingroup$ Yes - I wrote that the bias term is 0. My question is why there is variance for noise ($\epsilon$) and variance for our estimator $\hat y$ $\endgroup$
    – Jason
    Commented Apr 30, 2020 at 21:15
  • $\begingroup$ Hi: I still might not be understanding your question but in the unbiased case, the variance for the noise is the variance for the estimator $\hat{y}$. they're the same thing. . $\endgroup$
    – mlofton
    Commented May 1, 2020 at 13:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.