I am trying to prove a lemma. Consider $X_1,.,X_n$ to be i.i.d random variables. Show that, in particular: $$E\|\hat{\mu}-\mu \|^2=\frac{1}{n}\text{trace}(\Sigma)$$

I have the following partial solution:

$$E\|\hat{\mu}-\mu\|^2=E(\hat{\mu}-\mu)^T(\hat{\mu}-\mu)$$, since $\|x\|=x^Tx$, now using the definition of $\hat\mu=\frac{1}{n}\displaystyle\sum_{i=1}^nX_i$ I can write the last expression as: $$E(\frac{1}{n}\sum_{i=1}^n(X_i-\mu)^T(\frac{1}{n}\sum_{j=1}^nX_j-\mu))$$ Now, I put the factors $\frac{1}{n}$ outside the expectation $$\frac{1}{n^2}E(\sum_{i=1}^n(X_i-\mu)^T(\sum_{j=1}^n(X_j-\mu))$$

Since the random variables $X_i$ are i.i.d it means that in the cross products gotten solving the previous expression $i\neq j$, I will get something like $$\frac{1}{n^2}E(\sum_{i,j=1}^n(X_i-\mu)^T(X_j-\mu))$$

that is equal to: $$\frac{1}{n^2}(\sum_{i}^nE(X_i-\mu)^T(X_i-\mu))$$

now, according to the definition of matrix $\Sigma_{XY}=E[XY^T-\mu_X\mu_Y]=Cov(X,Y)$ in this case the last expression is equal to $Cov(X_i,X_i)=E[X_i^TX_i-\mu^t\mu]=Var(X_i)$ then the previous expression is equal to: $$=\frac{1}{n^2}\sum_{i=1}^nVar(X_i)=\frac{1}{n^2}trace(\Sigma)$$ That is not what I need to show, could you explain me what is wrong in my solution?

  • 1
    $\begingroup$ You are likely to find a much simpler solution by writing out the case $n=2$ in terms of the components of the vectors and matrices. $\endgroup$ – whuber Feb 1 '18 at 15:19
  • $\begingroup$ I cant conclude, could you help me: I consider $n=2$, then I got: $\frac{1}{4}(EX_1^T-\mu^T\mu+EX_2^TX_2-\mu^T\mu)$ that is equal to $\frac{1}{4}(Var(X_1)+Var(X_2))=\frac{1}{4}(\Sigma_{11}+\Sigma_{22})=\frac{1}{4}(trace(\Sigma))$ $\endgroup$ – Boris Feb 1 '18 at 15:26
  • $\begingroup$ I think a lot of people are giving great solutions. But the problem is you are confusion the sample size and the dimension size. $E(X_i-\mu)^\top (X_i-\mu)= trace(\Sigma)$ So you add up n trace of sigmas. $\endgroup$ – Josh Feb 1 '18 at 19:12

\begin{eqnarray} E||\hat\mu-\mu||^2 &=& E[(\hat\mu-\mu)^T(\hat\mu-\mu)] \\ &=& E[\textrm{trace}(\hat\mu-\mu)^T(\hat\mu-\mu)] \\ &=& E[\textrm{trace}(\hat\mu-\mu)(\hat\mu-\mu)^T] \\ &=& \textrm{trace}[E(\hat\mu-\mu)(\hat\mu-\mu)^T] \\ &=& \textrm{trace}\left[\frac{1}{n}\Sigma\right] \end{eqnarray}

The third line uses $\textrm{trace}(AB) = \textrm{trace}(BA)$ (when both products can be performed) and the fourth line the fact that the order of the expectation and trace operators can be interchanged.

  • $\begingroup$ But why,$$ (\hat\mu-\mu)^T(\hat\mu-\mu)=trace(\hat\mu-\mu)^T(\hat\mu-\mu)$$? $\endgroup$ – Boris Feb 1 '18 at 17:03
  • $\begingroup$ and why $E(\hat\mu-\mu)(\hat\mu-\mu)^T=\frac{1}{n}\Sigma$? That parts I dont have clear. $\endgroup$ – Boris Feb 1 '18 at 17:06
  • $\begingroup$ Boris, for your first question, it helps to think about the dimensions of the matrix, or to work out a simple example. $(\hat{\mu}-\mu)$ is a $n \times 1$ vector. Obviously, the transpose is $1 \times n$. And when you multiply the $1 \times n$ vector with a $n \times 1$ vector, you get $\sum_{i=1}^n$ terms, which is the same as the sum of the terms along the diagonal of the $n \times n$ matrix $(\hat{\mu} - \mu)^T(\hat{\mu} - \mu)$. And the sum of the diagonal is... the trace! Try it out with an example and you'll see. $\endgroup$ – A. G. Feb 1 '18 at 17:16
  • $\begingroup$ Doesn't the third line follow directly from the first more obviously than the second line? I wonder why you need the second line at all. You're using $x^T x = \text{trace}(x x^T),$ right? $\endgroup$ – Bridgeburners Feb 1 '18 at 19:09
  • $\begingroup$ Perhaps yes, I wanted to emphasize that a scalar is equal to its trace, as A.G. has explained in a previous comment. If that is clear, yes, one can proceed directly from the first to the third line. $\endgroup$ – F. Tusell Feb 1 '18 at 19:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.