Bias-variance tradeoff question From https://en.wikipedia.org/wiki/Bias%E2%80%93variance_tradeoff, the derivation for the bias-variance decomposition of the squared error is
$$
E[(y - \hat{f})^2] = Bias[\hat{f}]^2 + \sigma^2 + Var[\hat{f}]
$$
$f$ here is described as the true function, but I am confused what "true" function is in this sense. If we consider ordinary least squares, where $\hat{f} = X\hat{\beta}$, is $f = X\beta$, a linear function, or some true model that isn't necessarily linear?
I believe it is the latter, and not the former because if it is the former, then that would mean the bias error is zero since OLS is BLUE under the gauss-markov assumptions, which would reduce the above formula to
$$
E[(y - \hat{f})^2] = 0 + \sigma^2 + Var[\hat{f}]
$$
But this doesn't make sense because what if the distribution of $y$ isn't linear (for example, it could be some n-th order polynomial for n > 1), and we used a linear model $\hat{f}$ to fit the data? Then it is obvious that there is still bias error and it cannot possibly be zero, despite $\hat{f}$ being an unbiased estimator.
Hopefully my question makes sense, but if it doesn't, I'm basically thinking that $f$ is some true function that isn't necessarily linear. $\hat{f}$ is an approximation of $f$ and also of some function $f_{linear} = X\beta$. $f_{linear}$ is basically the best linear estimate of $f$. And $E[\hat{f}] = f_{linear}$, $E[\hat{f}] \neq f$.
 A: I think that's correct.
Take a look at section 7.3 The Bias–Variance Decomposition of Elements of Statistical Learning, Eqs. (7.13) and (7.14), which are formulated for ridge regression, but I believe the idea extends to unregularized OLS.
In the unconstrained optimization problem, I believe $f(X)$ is the true function, the same one you noted as not necessarily being linear.
You can see that the averaged squared bias is decomposed into a model bias and estimation bias term. I believe the estimation bias is zero for OLS if the Gauss-Markov assumptions are satisfied. The model bias is the bias induced by your linear model applied to data from possibly a non-linear function.
I think the term "bias" is sometimes ambiguously/loosely used. Consider https://en.wikipedia.org/wiki/Bias_of_an_estimator. The second sentence states "An estimator or decision rule with zero bias is called unbiased." I think "zero bias" here means "zero estimation bias" (Using ESL terminology), but not zero model bias.
I'm not 100% sure on this, so hopefully someone more experienced/knowledgeable than I am on this can comment.
