2
$\begingroup$

From https://en.wikipedia.org/wiki/Bias%E2%80%93variance_tradeoff, the derivation for the bias-variance decomposition of the squared error is

$$ E[(y - \hat{f})^2] = Bias[\hat{f}]^2 + \sigma^2 + Var[\hat{f}] $$

$f$ here is described as the true function, but I am confused what "true" function is in this sense. If we consider ordinary least squares, where $\hat{f} = X\hat{\beta}$, is $f = X\beta$, a linear function, or some true model that isn't necessarily linear?

I believe it is the latter, and not the former because if it is the former, then that would mean the bias error is zero since OLS is BLUE under the gauss-markov assumptions, which would reduce the above formula to

$$ E[(y - \hat{f})^2] = 0 + \sigma^2 + Var[\hat{f}] $$

But this doesn't make sense because what if the distribution of $y$ isn't linear (for example, it could be some n-th order polynomial for n > 1), and we used a linear model $\hat{f}$ to fit the data? Then it is obvious that there is still bias error and it cannot possibly be zero, despite $\hat{f}$ being an unbiased estimator.

Hopefully my question makes sense, but if it doesn't, I'm basically thinking that $f$ is some true function that isn't necessarily linear. $\hat{f}$ is an approximation of $f$ and also of some function $f_{linear} = X\beta$. $f_{linear}$ is basically the best linear estimate of $f$. And $E[\hat{f}] = f_{linear}$, $E[\hat{f}] \neq f$.

Improve this question
$\endgroup$
5
  • $\begingroup$ Even if it is an n-th order polynomial, we can still fit it with a linear model (en.wikipedia.org/wiki/Polynomial_regression). The true model, is the function behind the data generation process, and is generally not obtainable, we are trying to model it. We then try to analyze the solution from the best case point of view, if we would have known the true function. $\endgroup$
    – Gumeo
    Jun 25, 2020 at 18:52
  • $\begingroup$ @Gumeo I see, does that mean Iamanon's answer below is wrong then? My understanding is unbiased means $E[\hat{f}] = f_{\text{linear}} = X\beta + \epsilon$. But $f$, itself, may not actually be $f_{\text{linear}}$, so $E[\hat{f}] = f$ IFF $f = f_{\text{linear}}$ and $\hat{f}$ is unbiased. So in other words, I'm saying that if $E[\hat{f}] = f_{\text{linear}}$, then the bias term could still be non-zero, but I may be confusing something. $\endgroup$
    – David
    Jun 25, 2020 at 19:08
  • $\begingroup$ @Gumeo Also, sorry i mispoke in the OP. Rather than an n-th order polynomial, I meant what if the true model was a linear regression model, but a non-linear one. For example, what if the true model is such that it is an exponential function of the weights $\beta$? $\endgroup$
    – David
    Jun 25, 2020 at 19:19
  • $\begingroup$ The answer from lamanon is correct. But unbiased does not imply that f is a linear function. $\endgroup$
    – Gumeo
    Jun 26, 2020 at 7:02
  • $\begingroup$ Also if the model you fit is linear, then it still might be the wrong linear model, e.g. you do not pick all the correct variables for the model, so you could certainly have bias there. $\endgroup$
    – Gumeo
    Jun 26, 2020 at 7:03

1 Answer 1

Reset to default
1
$\begingroup$

I think that's correct.

Take a look at section 7.3 The Bias–Variance Decomposition of Elements of Statistical Learning, Eqs. (7.13) and (7.14), which are formulated for ridge regression, but I believe the idea extends to unregularized OLS.

In the unconstrained optimization problem, I believe $f(X)$ is the true function, the same one you noted as not necessarily being linear.

You can see that the averaged squared bias is decomposed into a model bias and estimation bias term. I believe the estimation bias is zero for OLS if the Gauss-Markov assumptions are satisfied. The model bias is the bias induced by your linear model applied to data from possibly a non-linear function.

I think the term "bias" is sometimes ambiguously/loosely used. Consider https://en.wikipedia.org/wiki/Bias_of_an_estimator. The second sentence states "An estimator or decision rule with zero bias is called unbiased." I think "zero bias" here means "zero estimation bias" (Using ESL terminology), but not zero model bias.

I'm not 100% sure on this, so hopefully someone more experienced/knowledgeable than I am on this can comment.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.