What effect does unobserved error and residuals have on the "bias" of a model? It's somewhat related to a question I posed earlier Different usage of the term "Bias" in stats/machine learning regarding the various usages of "bias."
I was the following questions a couple of months ago:

In simple linear regression, we have $Y = WX + b + e$ where $e$ is the standard normal error. What affect does $e$ have on the bias of the model? What if instead we have $Y = W(X + e) + b$?

When I was asked this question, I assumed they were asking about "bias" in terms of "bias"-variance tradeoff.
I know that the bias is an estimate is defined as
$$
\left( E[\hat{f}(x)] - f(x) \right)^2
$$
where $f$ is the true unobserved model and $\hat{f}$ is the model derived using linear regression. But $f$ and $\hat{f}$ aren't affected by the unobserved error $e$, so it seems in both situations, the unobserved error has no affect on the bias?
 A: $\newcommand{\e}{\varepsilon}\newcommand{\E}{\operatorname E}$In general $\hat f$ is a function of the data $X$ and $y$, and $y$ depends on $X$ and the error $\e$, so $\hat f$ itself has a functional dependency on $\e$.
In the usual setting of linear regression we have $y = X\beta + \e$ and
$$
\hat \beta = (X^TX)^{-1}X^Ty \\
= (X^TX)^{-1}X^T(X\beta + \e) \\
= \beta + (X^TX)^{-1}X^T\e
$$
so we can see exactly how they relate. For a particular point $x$ we'll have
$$
\hat f(x) = x^T\hat\beta = x^T\beta + x^T(X^TX)^{-1}X^T\e
$$
so if we make the usual assumptions of $\E[\e] = \mathbf 0$ and $\operatorname{Var}[\e] = \sigma^2 I$ then $\hat f(x)$ is a random variable centered at $x^T\beta$, so it is unbiased, and with a variance given by
$$
\operatorname{Var}[\hat f(x)] = x^T(X^TX)^{-1}X^T\operatorname{Var}[\e]X(X^TX)^{-1}x = \sigma^2 x^T(X^TX)^{-1}x.
$$
This shows how the uncertainty in our prediction for a particular point depends both on how that point $x$ relates to $X$ (the variance will be large when $x$ is mostly in the span of the bottom eigenvectors of the sample covariance matrix $X^TX$) and the underlying variance $\sigma^2$, which simply has a scaling effect here.
After taking expected values $\e$ no longer appears, but that's because the expected value is precisely integrating that out.
