2
$\begingroup$

The MSE can be decomposed as follows:

\begin{align*} \mathbb{E}\left[(\hat{\theta} - \theta)^2\right] &= \mathbb{E}\left[\left(\hat{\theta} - \mathbb{E}(\hat{\theta}) + \mathbb{E}(\hat{\theta}) - \theta\right)^2\right]\\ &= \mathbb{E}\left[\hat{\theta} - \mathbb{E}(\hat{\theta})\right]^2 + \mathbb{E}\left[\left(\mathbb{E}(\hat{\theta}) - \theta\right)^2\right] + 0\\ &= \mathbb{E}\left[\hat{\theta} - \mathbb{E}(\hat{\theta})\right]^2 + \left(\mathbb{E}(\hat{\theta}) - \theta\right)^2 \\ &= Var(\hat{\theta}) + Bias(\hat{\theta})^2 \end{align*}

Suppose $X$ is a random variable. Does a similar bias-variance decomposition exist for $\mathbb{E}\left[\left.(\hat{\theta} - \theta)^2\right|X\right]$?

\begin{align*} \mathbb{E}\left[\left.(\hat{\theta} - \theta)^2\right|X\right] &= \mathbb{E}\left[\left.\left(\hat{\theta} - \mathbb{E}(\hat{\theta}|X) + \mathbb{E}(\hat{\theta}|X) - \theta\right)^2\right|X\right]\\ &= \mathbb{E}\left[\left.\left(\hat{\theta}- \mathbb{E}(\hat{\theta}|X)\right)^2\right|X\right] + \mathbb{E}\left[\left.\left(\mathbb{E}(\hat{\theta}|X) - \theta\right)^2\right|X\right] + 2 \mathbb{E}\left[\left.\left(\hat{\theta} - \mathbb{E}(\hat{\theta}|X)\right)\left(\mathbb{E}(\hat{\theta}|X) - \theta\right)\right|X\right]\\ &= Var(\hat{\theta}|X) + \mathbb{E}\left[\left.\left(\mathbb{E}(\hat{\theta}|X) - \theta\right)^2\right|X\right] + 0\\ &= Var(\hat{\theta}|X) + \mathbb{E}\left[\left.\left(\mathbb{E}(\hat{\theta}|X) - \theta\right)^2\right|X\right] \end{align*}

Is the above correct?

Also, can the second term be written as the following?

\begin{align*}\mathbb{E}\left[\left.\left(\mathbb{E}(\hat{\theta}|X) - \theta\right)^2\right|X\right] &= \mathbb{E}\left[\left.\mathbb{E}(\hat{\theta}|X)^2 - 2\mathbb{E}(\hat{\theta}|X)\theta + \theta^2\right|X\right]\\ &= \mathbb{E}(\hat{\theta}|X)^2- 2\mathbb{E}(\hat{\theta}|X)\theta + \theta^2 \end{align*}

Putting it altogether, we have:

$$\mathbb{E}\left[\left.(\hat{\theta} - \theta)^2\right|X\right] = Var(\hat{\theta}|X)+\mathbb{E}(\hat{\theta}|X)^2- 2\mathbb{E}(\hat{\theta}|X)\theta + \theta^2$$

Edit:

By law of iterated expectation,

\begin{align*} \mathbb{E}\left[\left(\hat{\theta} - \theta\right)^2\right] &= \mathbb{E}\left[\mathbb{E}\left[\left.\left(\hat{\theta} - \theta\right)^2\right|X\right]\right]\\ &= \mathbb{E}\left[Var(\hat{\theta}|X)+\mathbb{E}(\hat{\theta}|X)^2- 2\mathbb{E}(\hat{\theta}|X)\theta + \theta^2\right]\\ &\overset{?}{=} Var(\hat{\theta}) + Bias(\hat{\theta})^2 \end{align*}

I'm not sure I see how the last line would hold?

$\endgroup$

2 Answers 2

2
$\begingroup$

Yes, it is correct. You can think of the original equation as a special case of the conditional one, where nothing is given. Your expansion of the second term is also correct. You can also sanity check the last equation by removing $X$'s as follows: $$\operatorname{Var}(\hat{\theta}|X)+\mathbb{E}(\hat{\theta}|X)^2- 2\mathbb{E}(\hat{\theta}|X)\theta + \theta^2\rightarrow \operatorname{Var}(\hat{\theta})+\underbrace{\mathbb{E}(\hat{\theta})^2- 2\mathbb{E}(\hat{\theta})\theta + \theta^2}_{\operatorname{Bias}(\hat\theta)^2}$$

$\endgroup$
2
  • $\begingroup$ Thanks. I have a follow up question: I'm not sure how to verify that $\mathbb{E}\left[\left(\hat{\theta} - \theta\right)^2\right] = \mathbb{E}\left[\mathbb{E}\left[\left.\left(\hat{\theta} - \theta\right)^2\right|X\right]\right]$ holds in this case. I've edited my original post. Do you have any pointers? $\endgroup$
    – Adrian
    Jan 17, 2021 at 17:45
  • $\begingroup$ Normally you don't have to show that some other way because you've done it already above, however total variance formula probably helps. $\endgroup$
    – gunes
    Jan 17, 2021 at 18:01
1
$\begingroup$

Suppose the estimator $\hat{\theta}$ is a function of random variable $X$ and denoted as $\hat{\theta} = g(X)$, then when $X=x$, $\hat{\theta} = g(x)$, or a constant value.

The term $\mathbb{E}(\hat{\theta}|X)$ introduced in the bias-variance decomposition is a function of $X$ which takes value $\mathbb{E}(\hat{\theta}=g(x)|X=x)=g(x)$ when $X=x$. Because of this, the function $\mathbb{E}(\hat{\theta}|X)$ is the same function as $\hat{\theta} = g(X)$. Similarly, the term $Var(\hat{\theta}|X)$ as a function of $X$, takes value $Var(\hat{\theta} = g(x) | X=x)=0$ when $X=x$. So $Var(\hat{\theta}|X)$ has a constant value of 0.

Using these information, introducing the term $\mathbb{E}(\hat{\theta}|X)$ seems to lead the expansion of $\mathbb{E}[(\hat{\theta}-\theta)^2|X]$ back to its original form.

If we instead introduce the term $\mathbb{E}(\theta|X)$ such that $\mathbb{E}[(\hat{\theta}-\theta)^2|X] = \mathbb{E}[(\hat{\theta}-\mathbb{E}(\theta|X)+\mathbb{E}(\theta|X)-\theta)^2|X] = ...$ after expanding the terms, you should be able to get $\mathbb{E}[(\hat{\theta}-\theta)^2|X] = \mathbb{E}[(\theta-\mathbb{E}(\theta|X))^2|X] + \mathbb{E}[(\mathbb{E}(\theta|X)-\hat{\theta})^2|X]$. Note that the first term here is just the conditional variance of $\theta$. By setting the estimator $\hat{\theta}=\mathbb{E}(\theta|X)$, the second term becomes 0, and the mean squared error given $X$ is minimized.

This answer https://stats.stackexchange.com/q/164391 may provide some more interpretations of the second term in the result $\mathbb{E}[(\mathbb{E}(\theta|X)-\hat{\theta})^2|X]$ and how it relates to bias-variance decomposition. From what I've seen before, the bias-variance decomposition is usually done under the classical inference.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.