1
$\begingroup$

I am solving the next exercise, but I have spent a lot of time and I can´t. For random vectors, $X_1,X_2,...\in\mathbb{R}^p$ The sample covariance with $\Sigma_1=0$ is given by: $$\hat\Sigma_n=\frac{1}{n-1}\sum_{i=1}^n(X_i-\hat{\mu_n})(X_i-\hat{\mu_n})^T$$ Prove that $\hat\Sigma_n=\frac{n-1}{n-2}\hat\Sigma_{n-1}+\frac{1}{n}\Delta_n\Delta_n^T$

Where $\Delta_n=X_n-\hat\mu_{n-1}$

I have the next part:

I start with the right side of the equality that I need to prove. i.e: $$=\frac{n-2}{n-1}(\frac{1}{n-2}\sum_{i=1}^{n-1}(X_i-\hat{\mu}_{n-1})(X_i-\hat{\mu}_{n-1})^T)+\frac{1}{n}\Delta_n\Delta_n^T+\frac{1}{n}\Delta_n\Delta_n^T\\ =\frac{1}{n-1}(\sum_{i=1}^{n-1}X_iX_i^T-\sum_{i=1}^{n-1}X_i\hat{\mu}_{n-1}^T-\hat{\mu}_{n-1}\sum_{i=1}^{n-1}X_i^T+\sum_{i=1}^{n-1}\hat{\mu}_{n-1}\hat{\mu}_{n-1}^T)+\frac{1}{n}\Delta_n\Delta_n^T$$ $$=\frac{1}{n-1}\sum_{i=1}^{n-1}X_iX_i^T-\hat{\mu}_{n-1}\hat{\mu}_{n-1}^T-\hat{\mu}_{n-1}\hat{\mu}_{n-1}^T+\hat{\mu}_{n-1}\hat{\mu}_{n-1}^T+\frac{1}{n}\Delta_n\Delta_n^T$$ In the previous line I used the fact that $\hat\mu_{n-1}=\frac{1}{n-1}\sum_{i=1}^{n-1}X_i$ then I conclude that $\sum_{i=1}^{n-1}X_i=(n-1)\hat\mu_{n-1}$ and with the common denominator it is simplified.

From here, I have replaced with the definition given of $\Delta_n$ and $\hat\mu_{n-1}=\frac{1}{n-1}\sum_{i=1}^{n-1}X_i$

But, I can´t conclude the exercise.

Additionally, I have a question related, At some stage of doing this proof I got something like that: $X_n\hat\mu_{n-1}^T$ and $\hat\mu_{n-1}X_n^T$, Since I am using vectors, this is a product of vectors and I got a matrix, but both are different or equal? according to me this is different because one is the transpose of other and is not equal, I did an example in R and I didn't get the same result, I mean I got that are different, Is ok my reasoning?

$\endgroup$
  • $\begingroup$ By working it out first for one-dimensional vectors you will see how the algebra goes. The solution will be identical for higher-dimensional vectors; indeed, by using vector and matrix notation you won't have to change anything. $\endgroup$ – whuber Oct 30 '17 at 15:11
4
$\begingroup$

It's easy if you write $$ \hat\Sigma_n= \frac{1}{n-1}\sum_{i=1}^nX_i X_i^T - \frac{n}{n-1}\hat{\mu}_n\hat{\mu}_n^T. $$ Split up the sum over $n$ elements into two parts. One will involve the first $n-1$ terms (which you can make look like the previous sample covariance), and the second will involve the most recent $n$th term. Finally, find a recursive formula for $\hat{\mu}_n$, and then plug that in.

Edit: The correct formula should be $$ \hat{\Sigma}_n = \frac{n-2}{n-1}\hat{\Sigma}_{n-1} + \frac{1}{n}\Delta_n \Delta_n^T. $$ I think you flipped that first fraction accidentally.

Here's a proof: \begin{align*} \hat{\Sigma}_n &= \frac{1}{n-1}\sum_{i=1}^n(x_i - \hat{\mu}_n)(x_i - \hat{\mu}_n)^T\\ &=\frac{1}{n-1}\sum_{i=1}^n x_ix_i^T - \frac{n}{n-1}\hat{\mu}_n\hat{\mu}_n^T\\ &= \frac{1}{n-1}\sum_{i=1}^{n-1} x_ix_i^T + \frac{1}{n-1}x_nx_n^T - \frac{n}{n-1}\hat{\mu}_n\hat{\mu}_n^T \\ &= \frac{1}{n-1}\sum_{i=1}^{n-1} x_ix_i^T + \frac{1}{n-1}x_nx_n^T - \frac{n}{n-1} \left(\frac{n-1}{n} \hat{\mu}_{n-1} + \frac{1}{n}x_n \right)\left(\frac{n-1}{n} \hat{\mu}_{n-1} + \frac{1}{n}x_n \right)^T \\ &= \frac{1}{n-1}\sum_{i=1}^{n-1} x_ix_i^T + \frac{1}{n-1}x_nx_n^T - \left( \hat{\mu}_{n-1} + \frac{1}{n-1}x_n \right)\left(\frac{n-1}{n} \hat{\mu}_{n-1} + \frac{1}{n}x_n \right)^T \\ &= \frac{1}{n-1} \sum_{i=1}^{n-1} x_ix_i^T + \frac{1}{n-1}x_nx_n^T - \frac{n-1}{n}\hat{\mu}_{n-1}\hat{\mu}_{n-1}^T - \frac{1}{n} \hat{\mu}_{n-1} x_n^T - \frac{1}{n}x_n\hat{\mu}_{n-1}^T - \frac{1}{(n-1)n} x_n x_n^T \\ &= \frac{1}{n-1} \sum_{i=1}^{n-1} x_ix_i^T + \frac{1}{n-1}x_nx_n^T - \frac{n-1}{n}\hat{\mu}_{n-1}\hat{\mu}_{n-1}^T - \frac{1}{n} \hat{\mu}_{n-1} x_n^T - \frac{1}{n}x_n\hat{\mu}_{n-1}^T + \frac{1}{n}x_n x_n^T - \frac{1}{n-1} x_n x_n^T \\ &= \frac{1}{n-1} \sum_{i=1}^{n-1} x_ix_i^T + \frac{1}{n-1}x_nx_n^T - \hat{\mu}_{n-1}\hat{\mu}_{n-1}^T - \frac{1}{n-1} x_n x_n^T +\left( \frac{1}{n}x_n x_n^T - \frac{1}{n}x_n\hat{\mu}_{n-1}^T - \frac{1}{n} \hat{\mu}_{n-1} x_n^T + \frac{1}{n}\hat{\mu}_{n-1}\hat{\mu}_{n-1}^T \right) \\ &= \frac{1}{n-1} \sum_{i=1}^{n-1} x_ix_i^T - \hat{\mu}_{n-1}\hat{\mu}_{n-1}^T + \frac{1}{n}\Delta_n\Delta_n^T \\ &= \frac{n-2}{n-1}\left(\frac{1}{n-2} \sum_{i=1}^{n-1} x_ix_i^T - \frac{n-1}{n-2}\hat{\mu}_{n-1}\hat{\mu}_{n-1}^T\right) + \frac{1}{n}\Delta_n\Delta_n^T \\ &= \frac{n-2}{n-1}\hat{\Sigma}_{n-1} + \frac{1}{n}\Delta_n \Delta_n^T. \end{align*}

$\endgroup$
  • 1
    $\begingroup$ You are right, but why the other way doesn't work? How to get the idea of solving in the other way?, why not solve suing the right side? $\endgroup$ – Boris Oct 30 '17 at 1:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.