23
$\begingroup$

In showing that MSE can be decomposed into variance plus the square of Bias, the proof in Wikipedia has a step, highlighted in the picture. How does this work? How is the expectation pushed in to the product from the 3rd step to the 4th step? If the two terms are independent, shouldn't the expectation be applied to both the terms? and if they aren't, is this step valid?enter image description here

$\endgroup$
22
$\begingroup$

The trick is that $\mathbb{E}(\hat{\theta}) - \theta$ is a constant.

$\endgroup$
  • 1
    $\begingroup$ Oh I see. The only unknown here is the estimator. Right? $\endgroup$ – statBeginner Nov 9 '14 at 19:43
  • 2
    $\begingroup$ Yes. Taking expectation means that the estimator goes to whatever it's estimating, that's what makes the $\mathbf{E}(\hat{\theta} - \mathbf{E}(\hat{\theta}))$ go to 0. $\endgroup$ – AdamO Nov 9 '14 at 23:38
  • 5
    $\begingroup$ Sorry, that sentence doesn't make much sense to me. If an estimator went to whatever it was estimating, wouldn't that make it unbiased? Can it be explained by saying $\mathbb{E}(\hat{\theta} - \mathbb{E}(\hat{\theta}))$ = $\mathbb{E}(\hat{\theta}) - \mathbb{E}(\mathbb{E}(\hat{\theta}))$ = $\mathbb{E}(\hat{\theta}) - \mathbb{E}(\hat{\theta})$ = 0 ? $\endgroup$ – user1158559 Apr 6 '17 at 21:16
  • $\begingroup$ @user1158559 the product term in the middle is a constant times something with expected value 0. Even if theta-hat is biased, it's still a constant times 0. $\endgroup$ – AdamO Apr 18 '17 at 15:49
  • 3
    $\begingroup$ $\mathbb{E}(\hat{\theta}) - \theta$ is a variable and not a constant. Also, the trick is less trivial and $\mathbb{E}(c)$ with $c$ a constant does not become 0 as the default (for instance $\mathbb{E}((\mathbb{E}(\hat{\theta}) - \theta)^2) \neq 0$). The real trick lies in the fact that $\int x p(x)$ is the constant (and can be taken out of an integral) so $\int (\int x p(x))p(x) = (\int x p(x)) \int p(x) = (\int x p(x)) 1 = (\int x p(x))$ $\endgroup$ – Martijn Weterings Sep 20 '17 at 17:32
4
$\begingroup$

Adam's answer is correct about the trick that $E(\hat{\theta}) - \theta$ is a constant. However it helps to find the end result, and does not clearly explain the question about the specific step in the wikipedia article (edit: which I see now was ambiguous being about the highlight and the step from line three to line four).

(note the question is about the variable $\mathbb{E}\left[\hat{\theta}\right] - \hat{\theta}$, which differs from constant $\mathbb{E}\left[\hat{\theta}\right] - \theta$ in the answer of Adam. I wrote this wrong in my comment. Expanding the terms for more clarity: the variable is the estimated $\hat{\theta}$, constants are the expectation of this estimate $\mathbb{E}\left[\hat{\theta}\right]$ and the true value $\theta$ )

Trick 1: Consider

the variable $x=\hat{\theta}$

the constant $a=\mathbb{E}\left[ \hat{\theta} \right]$

and the constant $b = \theta$

Then the relation can be written easily using the transformation rules expressing the moments of variable $x$ about $b$ in terms of the moments of variable $x$ about $a$.

$\mathbb{E}\left[(x-b)^n\right] = \sum_{i=0}^{n} \binom{n}{i}\mathbb{E}\left[(x-a)^i\right] (a-b)^{n-i}$

Trick 2: For the second moment the above formula has three terms in the summation. We can eliminate one of them (the case $i=1$) because $\mathbb{E}\left[(\hat{\theta}-\mathbb{E}\left[ \hat{\theta} \right])\right] = \mathbb{E}\left[ \hat{\theta} \right] - \mathbb{E}\left[\mathbb{E}\left[ \hat{\theta} \right]\right] = 0$

Here one can also make the argument with something being a constant. Namely $\mathbb{E}(a)=a$ if $a$ is a constant and using $a=\mathbb{E}(\theta)$, which is a constant, you get $\mathbb{E}(\mathbb{E}(\theta))=\mathbb{E}(\theta)$.

More intuitively: we made the moment of $x$ about $a$, equal to a central moment (and the odd central moments are zero). We get a bit of a tautology. By substracting the mean from the variable, $\hat{\theta}-\mathbb{E}\left[ \hat{\theta} \right]$, we generate a variable with mean zero. And, the mean of 'a variable with mean zero' is zero.


The wikipedia article uses these two tricks in respectively the third and fourth line.

  • The nested expectation in the third line

    $\mathbb{E}\left[(\hat{\theta} -\mathbb{E}(\hat{\theta}))(\mathbb{E}({\hat{\theta}})-\theta) \right]$

    is simplified by taking the constant part $(\mathbb{E}({\hat{\theta}})-\theta)$ outside of it (trick 1).

  • The term $\mathbb{E}\left(\hat{\theta} -\mathbb{E}(\hat{\theta})\right)$ is solved (as equal to zero) by using the fact that the variable $\hat{\theta} -\mathbb{E}(\hat{\theta})$ has mean zero (trick 2).

$\endgroup$
3
$\begingroup$

$E(\hat{\theta}) - \theta$ is not a constant.

The comment of @user1158559 is actually the correct one:

$$ E[\hat{\theta} - E(\hat{\theta})] = E(\hat{\theta}) - E[E(\hat{\theta})] = E(\hat{\theta}) - E(\hat{\theta}) = 0 $$

$\endgroup$
  • $\begingroup$ I don't see what you are trying to show. Also the bias may not be zero but that does not mean that it isn't a constant. $\endgroup$ – Michael Chernick Sep 21 '17 at 0:20
  • $\begingroup$ It is not a constant because $\hat{\theta} = f(D)$ where $D$ is a given training data, which is also a random variable. Thus, its expectation is not a constant. $\endgroup$ – little_monster Sep 21 '17 at 0:49
  • $\begingroup$ Also, the fact that it is not a constant or not cannot explain how step 4 is possible from step 3. On the other hand, the comment of @ user1158559 explains that. $\endgroup$ – little_monster Sep 21 '17 at 0:52
  • $\begingroup$ @Michael, there has been confusion about the question. The highlighted part contains this expression $\mathbb{E}(\hat{\theta} - \mathbb{E}(\hat{\theta}) )=0$, but in the text of the question it is mentioned that it is instead about the change from the third line to the fourth line, changing the nesting of expectations. $\endgroup$ – Martijn Weterings Sep 21 '17 at 15:14

protected by kjetil b halvorsen Aug 25 '17 at 8:55

Thank you for your interest in this question. Because it has attracted low-quality or spam answers that had to be removed, posting an answer now requires 10 reputation on this site (the association bonus does not count).

Would you like to answer one of these unanswered questions instead?

Not the answer you're looking for? Browse other questions tagged or ask your own question.