In showing that MSE can be decomposed into variance plus the square of Bias, the proof in Wikipedia has a step, highlighted in the picture. How does this work? How is the expectation pushed in to the product from the 3rd step to the 4th step? If the two terms are independent, shouldn't the expectation be applied to both the terms? and if they aren't, is this step valid?
$\begingroup$
$\endgroup$
1
Amelio Vazquez-Reina
19.5k27 gold badges81 silver badges120 bronze badges
asked Nov 9, 2014 at 19:28
-
$\begingroup$ For future reference, this is imo a pretty clear derivation of the decomposition, which is more explicit in notation. $\endgroup$– ngmirCommented Jul 26, 2023 at 16:30
Add a comment
|
3 Answers
$\begingroup$
$\endgroup$
10
The trick is that $\mathbb{E}(\hat{\theta}) - \theta$ is a constant.
-
1$\begingroup$ Oh I see. The only unknown here is the estimator. Right? $\endgroup$ Commented Nov 9, 2014 at 19:43
-
2$\begingroup$ Yes. Taking expectation means that the estimator goes to whatever it's estimating, that's what makes the $\mathbf{E}(\hat{\theta} - \mathbf{E}(\hat{\theta}))$ go to 0. $\endgroup$– AdamOCommented Nov 9, 2014 at 23:38
-
7$\begingroup$ Sorry, that sentence doesn't make much sense to me. If an estimator went to whatever it was estimating, wouldn't that make it unbiased? Can it be explained by saying $\mathbb{E}(\hat{\theta} - \mathbb{E}(\hat{\theta}))$ = $\mathbb{E}(\hat{\theta}) - \mathbb{E}(\mathbb{E}(\hat{\theta}))$ = $\mathbb{E}(\hat{\theta}) - \mathbb{E}(\hat{\theta})$ = 0 ? $\endgroup$ Commented Apr 6, 2017 at 21:16
-
$\begingroup$ @user1158559 the product term in the middle is a constant times something with expected value 0. Even if theta-hat is biased, it's still a constant times 0. $\endgroup$– AdamOCommented Apr 18, 2017 at 15:49
-
1$\begingroup$ Adam, you were right (+1), I was confused about the original question, because it mentions a highlight in the picture (which is about that term going to zero instead of taking the constant term outside the brackets of the expectation). So I wanted to talk about the variable $\mathbb{E}(\hat{\theta})-\hat{\theta}$, but instead I copied (because I was lazy) from your answer the constant term $\mathbb{E}(\hat{\theta})-\theta$ (and could not change it anymore after 5 minutes). Besides (1) using the wrongly copied term, I also (2) misunderstood the question. The maths used variabl $\hat{\theta}$ $\endgroup$ Commented Sep 21, 2017 at 16:28
$\begingroup$
$\endgroup$
2
There has been some confusion about the question which was ambiguous being about the highlight and the step from line three to line four.
There are two terms that look a lot like each other.
$$\mathbb{E}\left[\hat{\theta}\right] - \theta \quad \text{vs} \quad \mathbb{E}\left[\hat{\theta}\right] - \hat\theta$$
The question, about the step from 3rd to 4th line, relates to the first term:
$\mathbb{E}[\hat{\theta}] - \theta$ this is the bias for the estimator $\hat\theta$
The bias is the same (constant) value every time you take a sample, and because of that you can take it out of the expectation operator (so that is how the step from the 3rd to 4th line, taking the constant out, is done).
Note that you should not interpret this as a Bayesian analysis where $\theta$ is variable. It is a frequentist analysis which conditions on the parameters $\theta$. So we are computing more specifically $\mathbb{E}[(\hat{\theta} - \theta)^2 \vert \theta]$, the expectation value of the squared error conditional on $\theta$, instead of $\mathbb{E}[(\hat{\theta} - \theta)^2]$. This conditioning is often implied implicitly in a frequentist analysis.
The question about the highlighted expression is about the second term
$\mathbb{E}[\hat{\theta}] - \hat{\theta}$ this is the deviation from the mean for the estimator $\hat{\theta}$.
It's expectation value is also called the 1st central moment which is always zero (so that is how the highlighted step, putting the expectation equal to zero, is done).
answered Sep 21, 2017 at 14:41
-
$\begingroup$ This is an excellent answer. Thanks. Perhaps this is more trivial, but this question and explanation doesn't seem to explain how one truly gets the 4th expression from the 3rd. To not deviate too much from this particular question, I left a new one here $\endgroup$– JoshCommented May 30, 2020 at 16:40
-
1$\begingroup$ @Josh, I have now stressed it a bit more. $\endgroup$ Commented May 30, 2020 at 16:48
$\begingroup$
$\endgroup$
9
$E(\hat{\theta}) - \theta$ is not a constant.
The comment of @user1158559 is actually the correct one:
$$ E[\hat{\theta} - E(\hat{\theta})] = E(\hat{\theta}) - E[E(\hat{\theta})] = E(\hat{\theta}) - E(\hat{\theta}) = 0 $$
-
$\begingroup$ I don't see what you are trying to show. Also the bias may not be zero but that does not mean that it isn't a constant. $\endgroup$ Commented Sep 21, 2017 at 0:20
-
$\begingroup$ It is not a constant because $\hat{\theta} = f(D)$ where $D$ is a given training data, which is also a random variable. Thus, its expectation is not a constant. $\endgroup$ Commented Sep 21, 2017 at 0:49
-
3$\begingroup$ @Josh, yes $$\text{$E(\hat\theta) - \theta$, the bias of the estimator $\hat{\theta}$, is constant}$$ (in the sense that it is the same for each sample). $$\text{$E(\hat\theta) - \hat\theta$, the error of the estimator $\hat\theta$, is not a constant}$$For instance, say we wish to approximate the mean of a normal distributed population by using the median. Then the the bias is $E(\hat\theta) - \theta =0$ and constantly the same for every sample, but the estimate $\hat{\theta}$ is not the same for every sample, ie. $E(\hat\theta) - \hat\theta$ will be different each sample. $\endgroup$ Commented May 30, 2020 at 15:31
-
1$\begingroup$ @Josh, this answer is right about the step in the highlighted block, why the expectation value of the the difference of the estimator with it's expectation value, $E(\hat\theta)-\hat\theta$, which is variable, is zero (note: in the previous comment I wrongly refered to this as the error of the estimator, which is $\hat\theta - \theta$ instead). This answer is wrong about $E(\hat\theta)-\theta$ not being a constant. $\endgroup$ Commented May 30, 2020 at 15:41
-
1$\begingroup$ Of course, you must not consider this from a Bayesian point of view where $\theta$ is treated as variable, but you must consider it from a frequentist point of view which conditions on $\theta$. $\endgroup$ Commented May 30, 2020 at 15:44