2
$\begingroup$

Referring to: How to understand that MLE of variance is biased in a Gaussian distribution

at some point during calculation the formula of the sum of the expected value becomes a single expected value:

confusing formula

The explanation given is: With the last step following since due to $E[X^2_n]$ being equal across n due to coming from the same distribution.

I can't understand:

  1. What does the expectation of a value mean, in this case what is even $E[X_n]$
  2. Why should the expected value of an observation value be equal?
  3. What's the proof for $Var[1/N *...] = (1/N)^2 Var[...]$

Thanks for the help and sorry for the bad format I don't know how to write formulas, this is my first post xD

$\endgroup$
1
$\begingroup$

$X_n$ is a Gaussian RV for each $n$; since iid, they all have the same mean, variance, moments etc. i.e. simply we have $$E[X_1^2]=E[X_2^2]=\dots=E[X_n^2]$$

Therefore, $\frac{1}{N}\sum_{n=1}^N E[X_n^2]=\frac{1}{N}NE[X_1^2]=E[X_2^2]=E[X_n^2]$ for any $n$. This is why we can remove the summation.

The proof for scaling property of variance can be: $$\operatorname{var(cX)}=E[(cX)^2]-E[cX]^2=c^2(E[X^2]-E[X]^2)=c^2\operatorname{var}(X)$$

$\endgroup$
8
  • $\begingroup$ According to book though the Xn are the values of one observation, as stated: "we obtain the maximum likelihood solution which is the sample mean, i.e., the mean of the observed values {xn}" and "We first note that the maximum likelihood solutions µML and σ2ML are functions of the data set values x1,...,xN. Consider the expectations of these quantities with respect to the data set value". If this is the case the expected value shouldn't be different for each observation? $\endgroup$
    – Cristian
    Feb 10 '20 at 13:07
  • $\begingroup$ @Cristian This is the chicken & egg problem most novice practitioners are trapped in. $X_n$'s are observations sampled from a distribution. So, they're actually random variables o/w we wouldn't have sampled them and they wouldn't belong to a distribution. They're not fixed constants. Knowing (or pretending to know) and using them in our calculations doesn't mean that they're not RVs. In ML estimators, you pretend if you know $X_n$, use them in your estimator formulas, but when it comes to evaluating the bias of your estimator, we just take the expectation. How hypocrite of us :) $\endgroup$
    – gunes
    Feb 10 '20 at 13:14
  • $\begingroup$ Yeah I'm quite new to random processes and this is really confusing for me, so you say that even tough the values come from an observation we consider them as variables? How would the expectation formula change? would we have a constant in the integration or what? $\endgroup$
    – Cristian
    Feb 10 '20 at 13:20
  • $\begingroup$ In any expected value calculation, you don't deal with $x_n,X_n$ or etc. When you know the PDF, e.g. $x_n\sim f(x)$, the expected value can be written with any variable: $$E[x_n]=\int \tau f(\tau)d\tau$$ $\endgroup$
    – gunes
    Feb 10 '20 at 13:25
  • 1
    $\begingroup$ Wonderful I think I get it, so it's the same also for the variance for example, technically we could evaluate the variance of a single value (coming from a sample) but since we don't know the pdf we need multiple observation to attempt to determine it, right? $\endgroup$
    – Cristian
    Feb 10 '20 at 13:52
1
$\begingroup$

Note that $x_n$'s are random variables, not specific numbers. Assuming that $x_n$ is gaussian distributed with density $p$, it's expectation is per definition $$ E[x_n] = \int_\mathbb{R} x \cdot p(x) dx. $$ If we apply a function to $x_n$, it's expectation is $$E[f(x_n)] = \int_\mathbb{R} f(x) \cdot p(x) dx, $$ in your case $f(x) = x^2$.

The calculation you reference, assumes that the $x_n$ all come from the same distribution, i.e. they have the same density $p$ and thus the same expectation.

For your last question see e.g. here for a more general statement and proof.

$\endgroup$
3
  • $\begingroup$ According to book though the Xn are the values of one observation, as stated: "we obtain the maximum likelihood solution which is the sample mean, i.e., the mean of the observed values {xn}" and "We first note that the maximum likelihood solutions µML and σ2ML are functions of the data set values x1,...,xN. Consider the expectations of these quantities with respect to the data set value" $\endgroup$
    – Cristian
    Feb 10 '20 at 13:05
  • $\begingroup$ If this is the case the expected value shouldn't be different for each observation? $\endgroup$
    – Cristian
    Feb 10 '20 at 13:10
  • 1
    $\begingroup$ @Cristian It's not different for each observation because they're sampled from the same dist. $\endgroup$
    – gunes
    Feb 10 '20 at 13:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.