In my textbook, there is a statement mentioned on the topic of linear regression/machine learning, without a proof or rigorious justification, which is simply quoted as,

Consider a noisy target, $ y = (w^{*})^T \textbf{x} + \epsilon $, for generating the data, where $\epsilon$ is a noise term with zero mean and $\sigma^2$ variance, independently generated for every example $(\textbf{x},y)$. The expected error of the best possible linear fit to this target is thus $\sigma^2$.

Here my concerns are,

How do we know what the best possible linear fit is ? Assuming in some way we estimated best linear fit, is it $y = (w^{*})^T \textbf{x}$ ? If so why can not other parameter $w_2^{*}$ be the best fitting parameter ?


The cost function is convex so there is only one minimum. You can also think to the least square estimator as the projection of your on the columns of x. So the part of the y that you can not estimate is the epsilon


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.