Conditional bias-variance decomposition of MSE

Question

The MSE can be decomposed as follows:

\begin{align*} \mathbb{E}\left[(\hat{\theta} - \theta)^2\right] &= \mathbb{E}\left[\left(\hat{\theta} - \mathbb{E}(\hat{\theta}) + \mathbb{E}(\hat{\theta}) - \theta\right)^2\right]\\ &= \mathbb{E}\left[\hat{\theta} - \mathbb{E}(\hat{\theta})\right]^2 + \mathbb{E}\left[\left(\mathbb{E}(\hat{\theta}) - \theta\right)^2\right] + 0\\ &= \mathbb{E}\left[\hat{\theta} - \mathbb{E}(\hat{\theta})\right]^2 + \left(\mathbb{E}(\hat{\theta}) - \theta\right)^2 \\ &= Var(\hat{\theta}) + Bias(\hat{\theta})^2 \end{align*}

Suppose $X$ is a random variable. Does a similar bias-variance decomposition exist for $\mathbb{E}\left[\left.(\hat{\theta} - \theta)^2\right|X\right]$?

\begin{align*} \mathbb{E}\left[\left.(\hat{\theta} - \theta)^2\right|X\right] &= \mathbb{E}\left[\left.\left(\hat{\theta} - \mathbb{E}(\hat{\theta}|X) + \mathbb{E}(\hat{\theta}|X) - \theta\right)^2\right|X\right]\\ &= \mathbb{E}\left[\left.\left(\hat{\theta}- \mathbb{E}(\hat{\theta}|X)\right)^2\right|X\right] + \mathbb{E}\left[\left.\left(\mathbb{E}(\hat{\theta}|X) - \theta\right)^2\right|X\right] + 2 \mathbb{E}\left[\left.\left(\hat{\theta} - \mathbb{E}(\hat{\theta}|X)\right)\left(\mathbb{E}(\hat{\theta}|X) - \theta\right)\right|X\right]\\ &= Var(\hat{\theta}|X) + \mathbb{E}\left[\left.\left(\mathbb{E}(\hat{\theta}|X) - \theta\right)^2\right|X\right] + 0\\ &= Var(\hat{\theta}|X) + \mathbb{E}\left[\left.\left(\mathbb{E}(\hat{\theta}|X) - \theta\right)^2\right|X\right] \end{align*}

Is the above correct?

Also, can the second term be written as the following?

\begin{align*}\mathbb{E}\left[\left.\left(\mathbb{E}(\hat{\theta}|X) - \theta\right)^2\right|X\right] &= \mathbb{E}\left[\left.\mathbb{E}(\hat{\theta}|X)^2 - 2\mathbb{E}(\hat{\theta}|X)\theta + \theta^2\right|X\right]\\ &= \mathbb{E}(\hat{\theta}|X)^2- 2\mathbb{E}(\hat{\theta}|X)\theta + \theta^2 \end{align*}

Putting it altogether, we have:

$$\mathbb{E}\left[\left.(\hat{\theta} - \theta)^2\right|X\right] = Var(\hat{\theta}|X)+\mathbb{E}(\hat{\theta}|X)^2- 2\mathbb{E}(\hat{\theta}|X)\theta + \theta^2$$

Edit:

By law of iterated expectation,

\begin{align*} \mathbb{E}\left[\left(\hat{\theta} - \theta\right)^2\right] &= \mathbb{E}\left[\mathbb{E}\left[\left.\left(\hat{\theta} - \theta\right)^2\right|X\right]\right]\\ &= \mathbb{E}\left[Var(\hat{\theta}|X)+\mathbb{E}(\hat{\theta}|X)^2- 2\mathbb{E}(\hat{\theta}|X)\theta + \theta^2\right]\\ &\overset{?}{=} Var(\hat{\theta}) + Bias(\hat{\theta})^2 \end{align*}

I'm not sure I see how the last line would hold?

Answer 1

Yes, it is correct. You can think of the original equation as a special case of the conditional one, where nothing is given. Your expansion of the second term is also correct. You can also sanity check the last equation by removing $X$'s as follows: $$\operatorname{Var}(\hat{\theta}|X)+\mathbb{E}(\hat{\theta}|X)^2- 2\mathbb{E}(\hat{\theta}|X)\theta + \theta^2\rightarrow \operatorname{Var}(\hat{\theta})+\underbrace{\mathbb{E}(\hat{\theta})^2- 2\mathbb{E}(\hat{\theta})\theta + \theta^2}_{\operatorname{Bias}(\hat\theta)^2}$$

Answer 2

Suppose the estimator $\hat{\theta}$ is a function of random variable $X$ and denoted as $\hat{\theta} = g(X)$, then when $X=x$, $\hat{\theta} = g(x)$, or a constant value.

The term $\mathbb{E}(\hat{\theta}|X)$ introduced in the bias-variance decomposition is a function of $X$ which takes value $\mathbb{E}(\hat{\theta}=g(x)|X=x)=g(x)$ when $X=x$. Because of this, the function $\mathbb{E}(\hat{\theta}|X)$ is the same function as $\hat{\theta} = g(X)$. Similarly, the term $Var(\hat{\theta}|X)$ as a function of $X$, takes value $Var(\hat{\theta} = g(x) | X=x)=0$ when $X=x$. So $Var(\hat{\theta}|X)$ has a constant value of 0.

Using these information, introducing the term $\mathbb{E}(\hat{\theta}|X)$ seems to lead the expansion of $\mathbb{E}[(\hat{\theta}-\theta)^2|X]$ back to its original form.

If we instead introduce the term $\mathbb{E}(\theta|X)$ such that $\mathbb{E}[(\hat{\theta}-\theta)^2|X] = \mathbb{E}[(\hat{\theta}-\mathbb{E}(\theta|X)+\mathbb{E}(\theta|X)-\theta)^2|X] = ...$ after expanding the terms, you should be able to get $\mathbb{E}[(\hat{\theta}-\theta)^2|X] = \mathbb{E}[(\theta-\mathbb{E}(\theta|X))^2|X] + \mathbb{E}[(\mathbb{E}(\theta|X)-\hat{\theta})^2|X]$. Note that the first term here is just the conditional variance of $\theta$. By setting the estimator $\hat{\theta}=\mathbb{E}(\theta|X)$, the second term becomes 0, and the mean squared error given $X$ is minimized.

This answer https://stats.stackexchange.com/q/164391 may provide some more interpretations of the second term in the result $\mathbb{E}[(\mathbb{E}(\theta|X)-\hat{\theta})^2|X]$ and how it relates to bias-variance decomposition. From what I've seen before, the bias-variance decomposition is usually done under the classical inference.