From https://en.wikipedia.org/wiki/Bias%E2%80%93variance_tradeoff, the derivation for the bias-variance decomposition of the squared error is

$$E[(y - \hat{f})^2] = Bias[\hat{f}]^2 + \sigma^2 + Var[\hat{f}]$$

$$f$$ here is described as the true function, but I am confused what "true" function is in this sense. If we consider ordinary least squares, where $$\hat{f} = X\hat{\beta}$$, is $$f = X\beta$$, a linear function, or some true model that isn't necessarily linear?

I believe it is the latter, and not the former because if it is the former, then that would mean the bias error is zero since OLS is BLUE under the gauss-markov assumptions, which would reduce the above formula to

$$E[(y - \hat{f})^2] = 0 + \sigma^2 + Var[\hat{f}]$$

But this doesn't make sense because what if the distribution of $$y$$ isn't linear (for example, it could be some n-th order polynomial for n > 1), and we used a linear model $$\hat{f}$$ to fit the data? Then it is obvious that there is still bias error and it cannot possibly be zero, despite $$\hat{f}$$ being an unbiased estimator.

Hopefully my question makes sense, but if it doesn't, I'm basically thinking that $$f$$ is some true function that isn't necessarily linear. $$\hat{f}$$ is an approximation of $$f$$ and also of some function $$f_{linear} = X\beta$$. $$f_{linear}$$ is basically the best linear estimate of $$f$$. And $$E[\hat{f}] = f_{linear}$$, $$E[\hat{f}] \neq f$$.

• Even if it is an n-th order polynomial, we can still fit it with a linear model (en.wikipedia.org/wiki/Polynomial_regression). The true model, is the function behind the data generation process, and is generally not obtainable, we are trying to model it. We then try to analyze the solution from the best case point of view, if we would have known the true function. Jun 25 '20 at 18:52
• @Gumeo I see, does that mean Iamanon's answer below is wrong then? My understanding is unbiased means $E[\hat{f}] = f_{\text{linear}} = X\beta + \epsilon$. But $f$, itself, may not actually be $f_{\text{linear}}$, so $E[\hat{f}] = f$ IFF $f = f_{\text{linear}}$ and $\hat{f}$ is unbiased. So in other words, I'm saying that if $E[\hat{f}] = f_{\text{linear}}$, then the bias term could still be non-zero, but I may be confusing something. Jun 25 '20 at 19:08
• @Gumeo Also, sorry i mispoke in the OP. Rather than an n-th order polynomial, I meant what if the true model was a linear regression model, but a non-linear one. For example, what if the true model is such that it is an exponential function of the weights $\beta$? Jun 25 '20 at 19:19
• The answer from lamanon is correct. But unbiased does not imply that f is a linear function. Jun 26 '20 at 7:02
• Also if the model you fit is linear, then it still might be the wrong linear model, e.g. you do not pick all the correct variables for the model, so you could certainly have bias there. Jun 26 '20 at 7:03

In the unconstrained optimization problem, I believe $$f(X)$$ is the true function, the same one you noted as not necessarily being linear.