6
$\begingroup$

I am solving the next exercise, but I have spent a lot of time and I can´t. For random vectors, $X_1,X_2,...\in\mathbb{R}^p$ The sample covariance with $\Sigma_1=0$ is given by: $$\hat\Sigma_n=\frac{1}{n-1}\sum_{i=1}^n(X_i-\hat{\mu_n})(X_i-\hat{\mu_n})^T$$ Prove that $\hat\Sigma_n=\frac{n-1}{n-2}\hat\Sigma_{n-1}+\frac{1}{n}\Delta_n\Delta_n^T$

Where $\Delta_n=X_n-\hat\mu_{n-1}$

I have the next part:

I start with the right side of the equality that I need to prove. i.e: $$=\frac{n-2}{n-1}(\frac{1}{n-2}\sum_{i=1}^{n-1}(X_i-\hat{\mu}_{n-1})(X_i-\hat{\mu}_{n-1})^T)+\frac{1}{n}\Delta_n\Delta_n^T+\frac{1}{n}\Delta_n\Delta_n^T\\ =\frac{1}{n-1}(\sum_{i=1}^{n-1}X_iX_i^T-\sum_{i=1}^{n-1}X_i\hat{\mu}_{n-1}^T-\hat{\mu}_{n-1}\sum_{i=1}^{n-1}X_i^T+\sum_{i=1}^{n-1}\hat{\mu}_{n-1}\hat{\mu}_{n-1}^T)+\frac{1}{n}\Delta_n\Delta_n^T$$ $$=\frac{1}{n-1}\sum_{i=1}^{n-1}X_iX_i^T-\hat{\mu}_{n-1}\hat{\mu}_{n-1}^T-\hat{\mu}_{n-1}\hat{\mu}_{n-1}^T+\hat{\mu}_{n-1}\hat{\mu}_{n-1}^T+\frac{1}{n}\Delta_n\Delta_n^T$$ In the previous line I used the fact that $\hat\mu_{n-1}=\frac{1}{n-1}\sum_{i=1}^{n-1}X_i$ then I conclude that $\sum_{i=1}^{n-1}X_i=(n-1)\hat\mu_{n-1}$ and with the common denominator it is simplified.

From here, I have replaced with the definition given of $\Delta_n$ and $\hat\mu_{n-1}=\frac{1}{n-1}\sum_{i=1}^{n-1}X_i$

But, I can´t conclude the exercise.

Additionally, I have a question related, At some stage of doing this proof I got something like that: $X_n\hat\mu_{n-1}^T$ and $\hat\mu_{n-1}X_n^T$, Since I am using vectors, this is a product of vectors and I got a matrix, but both are different or equal? according to me this is different because one is the transpose of other and is not equal, I did an example in R and I didn't get the same result, I mean I got that are different, Is ok my reasoning?

$\endgroup$
1
  • $\begingroup$ By working it out first for one-dimensional vectors you will see how the algebra goes. The solution will be identical for higher-dimensional vectors; indeed, by using vector and matrix notation you won't have to change anything. $\endgroup$
    – whuber
    Commented Oct 30, 2017 at 15:11

1 Answer 1

11
$\begingroup$

It's easy if you write $$ \hat\Sigma_n= \frac{1}{n-1}\sum_{i=1}^nX_i X_i^T - \frac{n}{n-1}\hat{\mu}_n\hat{\mu}_n^T. $$ Split up the sum over $n$ elements into two parts. One will involve the first $n-1$ terms (which you can make look like the previous sample covariance), and the second will involve the most recent $n$th term. Finally, find a recursive formula for $\hat{\mu}_n$, and then plug that in.

Edit: The correct formula should be $$ \hat{\Sigma}_n = \frac{n-2}{n-1}\hat{\Sigma}_{n-1} + \frac{1}{n}\Delta_n \Delta_n^T. $$ I think you flipped that first fraction accidentally.

Here's a proof: \begin{align*} \hat{\Sigma}_n &= \frac{1}{n-1}\sum_{i=1}^n(x_i - \hat{\mu}_n)(x_i - \hat{\mu}_n)^T\\ &=\frac{1}{n-1}\sum_{i=1}^n x_ix_i^T - \frac{n}{n-1}\hat{\mu}_n\hat{\mu}_n^T\\ &= \frac{1}{n-1}\sum_{i=1}^{n-1} x_ix_i^T + \frac{1}{n-1}x_nx_n^T - \frac{n}{n-1}\hat{\mu}_n\hat{\mu}_n^T \\ &= \frac{1}{n-1}\sum_{i=1}^{n-1} x_ix_i^T + \frac{1}{n-1}x_nx_n^T - \frac{n}{n-1} \left(\frac{n-1}{n} \hat{\mu}_{n-1} + \frac{1}{n}x_n \right)\left(\frac{n-1}{n} \hat{\mu}_{n-1} + \frac{1}{n}x_n \right)^T \\ &= \frac{1}{n-1}\sum_{i=1}^{n-1} x_ix_i^T + \frac{1}{n-1}x_nx_n^T - \left( \hat{\mu}_{n-1} + \frac{1}{n-1}x_n \right)\left(\frac{n-1}{n} \hat{\mu}_{n-1} + \frac{1}{n}x_n \right)^T \\ &= \frac{1}{n-1} \sum_{i=1}^{n-1} x_ix_i^T + \frac{1}{n-1}x_nx_n^T - \frac{n-1}{n}\hat{\mu}_{n-1}\hat{\mu}_{n-1}^T - \frac{1}{n} \hat{\mu}_{n-1} x_n^T - \frac{1}{n}x_n\hat{\mu}_{n-1}^T - \frac{1}{(n-1)n} x_n x_n^T \\ &= \frac{1}{n-1} \sum_{i=1}^{n-1} x_ix_i^T + \frac{1}{n-1}x_nx_n^T - \frac{n-1}{n}\hat{\mu}_{n-1}\hat{\mu}_{n-1}^T - \frac{1}{n} \hat{\mu}_{n-1} x_n^T - \frac{1}{n}x_n\hat{\mu}_{n-1}^T + \frac{1}{n}x_n x_n^T - \frac{1}{n-1} x_n x_n^T \\ &= \frac{1}{n-1} \sum_{i=1}^{n-1} x_ix_i^T + \frac{1}{n-1}x_nx_n^T - \hat{\mu}_{n-1}\hat{\mu}_{n-1}^T - \frac{1}{n-1} x_n x_n^T +\left( \frac{1}{n}x_n x_n^T - \frac{1}{n}x_n\hat{\mu}_{n-1}^T - \frac{1}{n} \hat{\mu}_{n-1} x_n^T + \frac{1}{n}\hat{\mu}_{n-1}\hat{\mu}_{n-1}^T \right) \\ &= \frac{1}{n-1} \sum_{i=1}^{n-1} x_ix_i^T - \hat{\mu}_{n-1}\hat{\mu}_{n-1}^T + \frac{1}{n}\Delta_n\Delta_n^T \\ &= \frac{n-2}{n-1}\left(\frac{1}{n-2} \sum_{i=1}^{n-1} x_ix_i^T - \frac{n-1}{n-2}\hat{\mu}_{n-1}\hat{\mu}_{n-1}^T\right) + \frac{1}{n}\Delta_n\Delta_n^T \\ &= \frac{n-2}{n-1}\hat{\Sigma}_{n-1} + \frac{1}{n}\Delta_n \Delta_n^T. \end{align*}

$\endgroup$
1
  • 1
    $\begingroup$ You are right, but why the other way doesn't work? How to get the idea of solving in the other way?, why not solve suing the right side? $\endgroup$
    – Boris
    Commented Oct 30, 2017 at 1:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.