24
$\begingroup$

In showing that MSE can be decomposed into variance plus the square of Bias, the proof in Wikipedia has a step, highlighted in the picture. How does this work? How is the expectation pushed in to the product from the 3rd step to the 4th step? If the two terms are independent, shouldn't the expectation be applied to both the terms? and if they aren't, is this step valid?enter image description here

$\endgroup$
22
$\begingroup$

The trick is that $\mathbb{E}(\hat{\theta}) - \theta$ is a constant.

| cite | improve this answer | |
$\endgroup$
  • 1
    $\begingroup$ Oh I see. The only unknown here is the estimator. Right? $\endgroup$ – statBeginner Nov 9 '14 at 19:43
  • 2
    $\begingroup$ Yes. Taking expectation means that the estimator goes to whatever it's estimating, that's what makes the $\mathbf{E}(\hat{\theta} - \mathbf{E}(\hat{\theta}))$ go to 0. $\endgroup$ – AdamO Nov 9 '14 at 23:38
  • 5
    $\begingroup$ Sorry, that sentence doesn't make much sense to me. If an estimator went to whatever it was estimating, wouldn't that make it unbiased? Can it be explained by saying $\mathbb{E}(\hat{\theta} - \mathbb{E}(\hat{\theta}))$ = $\mathbb{E}(\hat{\theta}) - \mathbb{E}(\mathbb{E}(\hat{\theta}))$ = $\mathbb{E}(\hat{\theta}) - \mathbb{E}(\hat{\theta})$ = 0 ? $\endgroup$ – user1158559 Apr 6 '17 at 21:16
  • $\begingroup$ @user1158559 the product term in the middle is a constant times something with expected value 0. Even if theta-hat is biased, it's still a constant times 0. $\endgroup$ – AdamO Apr 18 '17 at 15:49
  • $\begingroup$ May i know why is E(θ^)−θ a constant? $\endgroup$ – Liting Cai Aug 23 '17 at 13:54
6
$\begingroup$

There has been some confusion about the question which was ambiguous being about the highlight and the step from line three to line four.

There are two terms that look a lot like each other.

$$\mathbb{E}\left[\hat{\theta}\right] - \theta \quad \text{vs} \quad \mathbb{E}\left[\hat{\theta}\right] - \hat\theta$$

The question, about the step from 3rd to 4th line, relates to the first term:

  • $\mathbb{E}[\hat{\theta}] - \theta$ this is the bias for the estimator $\hat\theta$

    The bias is the same (constant) value every time you take a sample, and because of that you can take it out of the expectation operator (so that is how the step from the 3rd to 4th line, taking the constant out, is done).

    Note that you should not interpret this as a Bayesian analysis where $\theta$ is variable. It is a frequentist analysis which conditions on the parameters $\theta$. So we are computing more specifically $\mathbb{E}[(\hat{\theta} - \theta)^2 \vert \theta]$, the expectation value of the squared error conditional on $\theta$, instead of $\mathbb{E}[(\hat{\theta} - \theta)^2]$. This conditioning is often implied implicitly in a frequentist analysis.

The question about the highlighted expression is about the second term

  • $\mathbb{E}[\hat{\theta}] - \hat{\theta}$ this is the deviation from the mean for the estimator $\hat{\theta}$.

    It's expectation value is also called the 1st central moment which is always zero (so that is how the highlighted step, putting the expectation equal to zero, is done).

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ This is an excellent answer. Thanks. Perhaps this is more trivial, but this question and explanation doesn't seem to explain how one truly gets the 4th expression from the 3rd. To not deviate too much from this particular question, I left a new one here $\endgroup$ – Josh May 30 at 16:40
  • 1
    $\begingroup$ @Josh, I have now stressed it a bit more. $\endgroup$ – Sextus Empiricus May 30 at 16:48
2
$\begingroup$

$E(\hat{\theta}) - \theta$ is not a constant.

The comment of @user1158559 is actually the correct one:

$$ E[\hat{\theta} - E(\hat{\theta})] = E(\hat{\theta}) - E[E(\hat{\theta})] = E(\hat{\theta}) - E(\hat{\theta}) = 0 $$

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ I don't see what you are trying to show. Also the bias may not be zero but that does not mean that it isn't a constant. $\endgroup$ – Michael R. Chernick Sep 21 '17 at 0:20
  • $\begingroup$ It is not a constant because $\hat{\theta} = f(D)$ where $D$ is a given training data, which is also a random variable. Thus, its expectation is not a constant. $\endgroup$ – little_monster Sep 21 '17 at 0:49
  • $\begingroup$ Also, the fact that it is not a constant or not cannot explain how step 4 is possible from step 3. On the other hand, the comment of @ user1158559 explains that. $\endgroup$ – little_monster Sep 21 '17 at 0:52
  • 1
    $\begingroup$ @Josh, yes $$\text{$E(\hat\theta) - \theta$, the bias of the estimator $\hat{\theta}$, is constant}$$ (in the sense that it is the same for each sample). $$\text{$E(\hat\theta) - \hat\theta$, the error of the estimator $\hat\theta$, is not a constant}$$For instance, say we wish to approximate the mean of a normal distributed population by using the median. Then the the bias is $E(\hat\theta) - \theta =0$ and constantly the same for every sample, but the estimate $\hat{\theta}$ is not the same for every sample, ie. $E(\hat\theta) - \hat\theta$ will be different each sample. $\endgroup$ – Sextus Empiricus May 30 at 15:31
  • 1
    $\begingroup$ Of course, you must not consider this from a Bayesian point of view where $\theta$ is treated as variable, but you must consider it from a frequentist point of view which conditions on $\theta$. $\endgroup$ – Sextus Empiricus May 30 at 15:44

Not the answer you're looking for? Browse other questions tagged or ask your own question.