Bias variance tradeoff when the estimated target function is also random

Question

I'm interested to understand "bias variance tradeoff" notion in a different setting than usually presented. In a setting where target $f$ (see the map $f$ below) is a random map rather than deterministic. So this is akin to Bayesian approach where our uncertainty in the target function is interpreted as some prior probability density over hypothesis space $\mathcal{H}$.

Here is how I understand the usual bias variance decomposition (usual, i.e. $f$ is deterministic):

Bias variance decomposition: Let an arbitrary deterministic map $f$ on an inner product space $\mathcal{H}$. Let an arbitrary random map $\widehat{f}_\mathcal{G} \in \mathcal{H}$ distributed according to some distribution $\mathcal{G}$ on $\mathcal{H}$, then

$$ \mathbb{E}_{\mathcal{G}}\left[\|f - \widehat{f}_\mathcal{G}\|^2\right] = \mathbb{E}_{\mathcal{G}} \left[\|\widehat{f}_\mathcal{G}\|^2\right] - \|\mathbb{E}_{\mathcal{G}}\widehat{f}_\mathcal{G}\|^2 + \|f - \mathbb{E}_{\mathcal{G}}\widehat{f}_\mathcal{G}\|^2 \\ = \operatorname{Var}(\widehat{f}_\mathcal{G})^2 + \operatorname{Bias}(f, \widehat{f}_\mathcal{G})^2 $$

Now, replace deterministic $f$ with a random function $f_{\mathcal{G}}$, such that $f_{\mathcal{G}}$ and $\widehat{f}_{\mathcal{G}}$ are dependent. In that case I can obtain

$$\mathbb{E}_{\mathcal{G}}\left[\|f_\mathcal{G} - \widehat{f}_\mathcal{G}\|^2\right] = \operatorname{Var}(\widehat{f}_\mathcal{G})^2 + \mathbb{E}_{\mathcal{G}}\left[\|f_\mathcal{G} - \mathbb{E}_{\mathcal{G}}\widehat{f}_\mathcal{G}\|^2\right] + 2(\langle \mathbb{E}_\mathcal{G} f_\mathcal{G}, \mathbb{E}_\mathcal{G} \widehat{f}_\mathcal{G} \rangle - \mathbb{E}_\mathcal{G}\langle f_\mathcal{G},\widehat{f}_\mathcal{G} \rangle ) $$

It seems to me that there is no longer a clear decomposition of error into bias and variance parts.

My question: Nevertheless is there some alternative interpretation of the above decomposition? In particular, what can we say about: $$ 2(\langle \mathbb{E}_\mathcal{G} f_\mathcal{G}, \mathbb{E}_\mathcal{G} \widehat{f}_\mathcal{G} \rangle - \mathbb{E}_\mathcal{G}\langle f_\mathcal{G},\widehat{f}_\mathcal{G} \rangle )~~?$$ does it have a knwon interpretation?

Would be glad to hear any comments. Hope the question is not too vague.

---

---

Proof of bias variance decomposition ($f$ deterministic) \begin{align*} \mathbb{E}_{\mathcal{G}}\left[\|f - \widehat{f}_\mathcal{G}\|^2\right] &= \mathbb{E}_{\mathcal{G}} \langle f- \widehat{f}_\mathcal{G}, f- \widehat{f}_\mathcal{G} \rangle \\ &= \| f\|^2 + \mathbb{E}_{\mathcal{G}} \left[\|\widehat{f}_\mathcal{G}\|^2 \right]- \mathbb{E}_{\mathcal{G}} \langle f, \widehat{f}_\mathcal{G}\rangle - \mathbb{E}_{\mathcal{G}} \langle \widehat{f}_\mathcal{G}, f \rangle \\ &= \underbrace{\mathbb{E}_{\mathcal{G}} \left[\|\widehat{f}_\mathcal{G}\|^2\right] - \|\mathbb{E}_{\mathcal{G}}\widehat{f}_\mathcal{G}\|^2}_{\operatorname{Var}(\widehat{f}_\mathcal{G})^2} \\ &\quad\quad\quad+ \underbrace{\| f\|^2 + \|\mathbb{E}_{\mathcal{G}}\widehat{f}_\mathcal{G}\|^2 - \mathbb{E}_{\mathcal{G}} \langle f, \widehat{f}_\mathcal{G}\rangle - \mathbb{E}_{\mathcal{G}} \langle \widehat{f}_\mathcal{G}, f \rangle}_{\mathbb{E}_{\mathcal{G}}\left[\|f - \mathbb{E}_{\mathcal{G}}\widehat{f}_\mathcal{G}\|^2\right] = \|f - \mathbb{E}_{\mathcal{G}}\widehat{f}_\mathcal{G}\|^2}\\ &= \operatorname{Var}(\widehat{f}_\mathcal{G})^2 + \operatorname{Bias}(f, \widehat{f}_\mathcal{G})^2 \end{align*}

---

Bias variance decomposition ($f_\mathcal{G}$ random, and $\widehat{f}_{\mathcal{G}}$ depends on $f_\mathcal{G}$)

\begin{align*} \mathbb{E}_{\mathcal{G}}\left[\|f_\mathcal{G} - \widehat{f}_\mathcal{G}\|^2\right] &= \mathbb{E}_{\mathcal{G}} \langle f_\mathcal{G} - \widehat{f}_\mathcal{G}, f_\mathcal{G} - \widehat{f}_\mathcal{G} \rangle \\ &= \mathbb{E}_{\mathcal{G}} \| f_\mathcal{G}\|^2 + \mathbb{E}_{\mathcal{G}} \left[\|\widehat{f}_\mathcal{G}\|^2 \right]- \mathbb{E}_{\mathcal{G}} \langle f_\mathcal{G}, \widehat{f}_\mathcal{G}\rangle - \mathbb{E}_{\mathcal{G}} \langle \widehat{f}_\mathcal{G}, f_\mathcal{G} \rangle \\ &= \underbrace{\mathbb{E}_{\mathcal{G}} \left[\|\widehat{f}_\mathcal{G}\|^2\right] - \|\mathbb{E}_{\mathcal{G}}\widehat{f}_\mathcal{G}\|^2}_{\operatorname{Var}(\widehat{f}_\mathcal{G})^2} \\ &+ \underbrace{ \mathbb{E}_\mathcal{G}\| f_\mathcal{G}\|^2 + \|\mathbb{E}_{\mathcal{G}}\widehat{f}_\mathcal{G}\|^2 - \mathbb{E}_{\mathcal{G}} \langle f_\mathcal{G}, \mathbb{E}_{\mathcal{G}} \widehat{f}_\mathcal{G}\rangle - \mathbb{E}_{\mathcal{G}} \langle \widehat{f}_\mathcal{G}, \mathbb{E}_{\mathcal{G}} f_\mathcal{G} \rangle}_{\mathbb{E}_{\mathcal{G}}\left[\|f_\mathcal{G} - \mathbb{E}_{\mathcal{G}}\widehat{f}_\mathcal{G}\|^2\right]}\\ &+ \mathbb{E}_{\mathcal{G}} \langle f_\mathcal{G}, \mathbb{E}_{\mathcal{G}} \widehat{f}_\mathcal{G}\rangle + \mathbb{E}_{\mathcal{G}} \langle \mathbb{E}_{\mathcal{G}} \widehat{f}_\mathcal{G}, f_\mathcal{G} \rangle - \mathbb{E}_{\mathcal{G}} \langle f_\mathcal{G}, \widehat{f}_\mathcal{G}\rangle - \mathbb{E}_{\mathcal{G}} \langle \widehat{f}_\mathcal{G}, f_\mathcal{G} \rangle \\ &= \operatorname{Var}(\widehat{f}_\mathcal{G})^2 + \operatorname{Bias}(f_\mathcal{G}, \widehat{f}_\mathcal{G})^2 + 2(\langle \mathbb{E}_\mathcal{G} f_\mathcal{G}, \mathbb{E}_\mathcal{G} \widehat{f}_\mathcal{G} \rangle - \mathbb{E}_\mathcal{G}\langle f_\mathcal{G},\widehat{f}_\mathcal{G} \rangle ) \end{align*}

Answer 1

2nd raw moment as sum of variance and mean

Any 2nd raw moment can be decomposed into the mean and central 2nd moment.

$$E[q^2] = E[q]^2 + E[(q-E[q])^2]$$

When the quantity $q$ is a difference between two quantities, $q = \hat{f}-f$, then you get

$$E[(\hat{f}-f)^2] = E[\hat{f}-f]^2 + E[(\hat{f}-f-E[\hat{f}-f])^2]$$

These two terms have a simple meaning in the context of regression where $f$ is some true value and (which is a constant), and where $\hat{f}$ is an estimate of $f$ (which varies from sample to sample).

• Mean of error $\mathbf{E[\hat{f}-f]}$ is the average of the difference between $\hat{f}$ and $f$. If $\hat{f}$ is on average different from $f$, then there is a bias. It is the part of the error that is not due to a variation that makes $\hat{f}$ consistently different from $f$.
• Variance of error $\mathbf{E[(\hat{f}-f-E[\hat{f}-f])^2]}$ is the variance of $\hat{f}-f$. When $f$ is constant then $$\text{Var}[\hat{f}-f] = \text{Var}[\hat{f}]$$

---

Toy model

If $f$ is a variable then the interpretation becomes slightly different. Let's look at a toy model to see how we can interpret it.

Model example: We consider regression estimating the expectation value $E[y(x)]$, where $y$ is a function of $x$ with some added noise

$$\begin{array}{} x &=& \lbrace 0,1,2,3,4,5,6,7,8,9,10 \rbrace\\ y &=& 1 + x + 0.2*(x-2)(x-5)(x-8) + \epsilon \quad\quad \text{where } \epsilon \sim N(0,1) \end{array}$$

The estimate is made by using simple linear regression, and it is done for some point $x_{target}$ which is uniformly distributed between $0$ and $10$.

The image below shows how the estimate depends on the particular targeted value.

What we see is how the bias and variance is not every time the same. Depending on the value of the true value of $f$ you can have different bias and different variance. The variance increases towards the edges, and the bias follows some S-shaped pattern (which is due to modeling a 3-rd order polynomial with a straight line).

Interpretation of bias-variance

The toy model above gives an example of a case where $f$ is variable. Now this time we have

• Mean of error $\mathbf{E[\hat{f}-f]}$, this remains the average difference between $\hat{f}$ and $f$. But now you could see the bias as a variable as well. It depends from case to case. However, you can see it as the average bias.

• Variance of error $\mathbf{E[(\hat{f}-f-E[\hat{f}-f])^2]}$, this variance could be described as $$\text{Var}[\hat{f}-f] = \text{Var}[\hat{f}] + \text{Var}[f] -2\text{Cov}[\hat{f}-f]$$ But in the context of the previous toy model these terms make little sense.

An alternative description would be to consider the variance conditional on $f$ and the law of total variance.

$$\begin{array}{rclcl} \text{Var}[\hat{f}-f] &=& E[\text{Var}[\hat{f}-f|f]] &+& \text{Var}[E[\hat{f}-f|f]]\\ &=& E[\text{Var}[\hat{f}|f]] &+& \text{Var}[E[\hat{f}-f|f]]\end{array}$$

So we could use:

$$E[(\hat{f}-f)^2] = {\underbrace{E[\hat{f} - f]}_{\text{average bias}}}^2 + \underbrace{E[\text{Var}[\hat{f}|f]]}_{\text{average estimator variance}} + \underbrace{\text{Var}[E[\hat{f}-f|f]]}_{\text{variance of bias}}$$

this third term is new.

The image below relating to the toy model above might give an intuition about these terms. In the right image, we consider the marginal distribution of the error term. We can consider this error term as being a function of

$$\text{error} = \text{mean bias} + (\text{bias} - \text{mean bias}) + (\text{error} - \text{bias})$$