Conditional bias-variance decomposition of MSE

The MSE can be decomposed as follows:

\begin{align*} \mathbb{E}\left[(\hat{\theta} - \theta)^2\right] &= \mathbb{E}\left[\left(\hat{\theta} - \mathbb{E}(\hat{\theta}) + \mathbb{E}(\hat{\theta}) - \theta\right)^2\right]\\ &= \mathbb{E}\left[\hat{\theta} - \mathbb{E}(\hat{\theta})\right]^2 + \mathbb{E}\left[\left(\mathbb{E}(\hat{\theta}) - \theta\right)^2\right] + 0\\ &= \mathbb{E}\left[\hat{\theta} - \mathbb{E}(\hat{\theta})\right]^2 + \left(\mathbb{E}(\hat{\theta}) - \theta\right)^2 \\ &= Var(\hat{\theta}) + Bias(\hat{\theta})^2 \end{align*}

Suppose $$X$$ is a random variable. Does a similar bias-variance decomposition exist for $$\mathbb{E}\left[\left.(\hat{\theta} - \theta)^2\right|X\right]$$?

\begin{align*} \mathbb{E}\left[\left.(\hat{\theta} - \theta)^2\right|X\right] &= \mathbb{E}\left[\left.\left(\hat{\theta} - \mathbb{E}(\hat{\theta}|X) + \mathbb{E}(\hat{\theta}|X) - \theta\right)^2\right|X\right]\\ &= \mathbb{E}\left[\left.\left(\hat{\theta}- \mathbb{E}(\hat{\theta}|X)\right)^2\right|X\right] + \mathbb{E}\left[\left.\left(\mathbb{E}(\hat{\theta}|X) - \theta\right)^2\right|X\right] + 2 \mathbb{E}\left[\left.\left(\hat{\theta} - \mathbb{E}(\hat{\theta}|X)\right)\left(\mathbb{E}(\hat{\theta}|X) - \theta\right)\right|X\right]\\ &= Var(\hat{\theta}|X) + \mathbb{E}\left[\left.\left(\mathbb{E}(\hat{\theta}|X) - \theta\right)^2\right|X\right] + 0\\ &= Var(\hat{\theta}|X) + \mathbb{E}\left[\left.\left(\mathbb{E}(\hat{\theta}|X) - \theta\right)^2\right|X\right] \end{align*}

Is the above correct?

Also, can the second term be written as the following?

\begin{align*}\mathbb{E}\left[\left.\left(\mathbb{E}(\hat{\theta}|X) - \theta\right)^2\right|X\right] &= \mathbb{E}\left[\left.\mathbb{E}(\hat{\theta}|X)^2 - 2\mathbb{E}(\hat{\theta}|X)\theta + \theta^2\right|X\right]\\ &= \mathbb{E}(\hat{\theta}|X)^2- 2\mathbb{E}(\hat{\theta}|X)\theta + \theta^2 \end{align*}

Putting it altogether, we have:

$$\mathbb{E}\left[\left.(\hat{\theta} - \theta)^2\right|X\right] = Var(\hat{\theta}|X)+\mathbb{E}(\hat{\theta}|X)^2- 2\mathbb{E}(\hat{\theta}|X)\theta + \theta^2$$

Edit:

By law of iterated expectation,

\begin{align*} \mathbb{E}\left[\left(\hat{\theta} - \theta\right)^2\right] &= \mathbb{E}\left[\mathbb{E}\left[\left.\left(\hat{\theta} - \theta\right)^2\right|X\right]\right]\\ &= \mathbb{E}\left[Var(\hat{\theta}|X)+\mathbb{E}(\hat{\theta}|X)^2- 2\mathbb{E}(\hat{\theta}|X)\theta + \theta^2\right]\\ &\overset{?}{=} Var(\hat{\theta}) + Bias(\hat{\theta})^2 \end{align*}

I'm not sure I see how the last line would hold?

Yes, it is correct. You can think of the original equation as a special case of the conditional one, where nothing is given. Your expansion of the second term is also correct. You can also sanity check the last equation by removing $$X$$'s as follows: $$\operatorname{Var}(\hat{\theta}|X)+\mathbb{E}(\hat{\theta}|X)^2- 2\mathbb{E}(\hat{\theta}|X)\theta + \theta^2\rightarrow \operatorname{Var}(\hat{\theta})+\underbrace{\mathbb{E}(\hat{\theta})^2- 2\mathbb{E}(\hat{\theta})\theta + \theta^2}_{\operatorname{Bias}(\hat\theta)^2}$$

• Thanks. I have a follow up question: I'm not sure how to verify that $\mathbb{E}\left[\left(\hat{\theta} - \theta\right)^2\right] = \mathbb{E}\left[\mathbb{E}\left[\left.\left(\hat{\theta} - \theta\right)^2\right|X\right]\right]$ holds in this case. I've edited my original post. Do you have any pointers? Commented Jan 17, 2021 at 17:45
• Normally you don't have to show that some other way because you've done it already above, however total variance formula probably helps. Commented Jan 17, 2021 at 18:01

Suppose the estimator $$\hat{\theta}$$ is a function of random variable $$X$$ and denoted as $$\hat{\theta} = g(X)$$, then when $$X=x$$, $$\hat{\theta} = g(x)$$, or a constant value.

The term $$\mathbb{E}(\hat{\theta}|X)$$ introduced in the bias-variance decomposition is a function of $$X$$ which takes value $$\mathbb{E}(\hat{\theta}=g(x)|X=x)=g(x)$$ when $$X=x$$. Because of this, the function $$\mathbb{E}(\hat{\theta}|X)$$ is the same function as $$\hat{\theta} = g(X)$$. Similarly, the term $$Var(\hat{\theta}|X)$$ as a function of $$X$$, takes value $$Var(\hat{\theta} = g(x) | X=x)=0$$ when $$X=x$$. So $$Var(\hat{\theta}|X)$$ has a constant value of 0.

Using these information, introducing the term $$\mathbb{E}(\hat{\theta}|X)$$ seems to lead the expansion of $$\mathbb{E}[(\hat{\theta}-\theta)^2|X]$$ back to its original form.

If we instead introduce the term $$\mathbb{E}(\theta|X)$$ such that $$\mathbb{E}[(\hat{\theta}-\theta)^2|X] = \mathbb{E}[(\hat{\theta}-\mathbb{E}(\theta|X)+\mathbb{E}(\theta|X)-\theta)^2|X] = ...$$ after expanding the terms, you should be able to get $$\mathbb{E}[(\hat{\theta}-\theta)^2|X] = \mathbb{E}[(\theta-\mathbb{E}(\theta|X))^2|X] + \mathbb{E}[(\mathbb{E}(\theta|X)-\hat{\theta})^2|X]$$. Note that the first term here is just the conditional variance of $$\theta$$. By setting the estimator $$\hat{\theta}=\mathbb{E}(\theta|X)$$, the second term becomes 0, and the mean squared error given $$X$$ is minimized.

This answer https://stats.stackexchange.com/q/164391 may provide some more interpretations of the second term in the result $$\mathbb{E}[(\mathbb{E}(\theta|X)-\hat{\theta})^2|X]$$ and how it relates to bias-variance decomposition. From what I've seen before, the bias-variance decomposition is usually done under the classical inference.