I watched Andrew NG's lecture on PCA and happened to come across this formula for computing Covariance Matrix which I don't comprehend . enter image description here

I feel there's no need for a summation(sigma) over ranging from i=1 to n , when that purpose is served by the 'Transpose term of x(i) ' .

I've seen formulae like this : enter image description here

where Covariance of a matrix Z is computed by just Z.Z' without the need for summation .

So , if both the formulae are computing the covariance matrix , I don't understand why their formula differ . Why do we need a summation ?

Can someone breakdown the formulae ( perhaps with an example) to help me understand if there are any differences between the two ? Both are formulae of covariance matrix.

  • 2
    $\begingroup$ X.X^T != (x^(i)) (x^(i))^T $\endgroup$ – Yahya Jun 28 at 17:02
  • $\begingroup$ Yes ! But if we didn't use the summation , It would just be X.X^T right ? I don't understand the need for summation . Other Covariance Matrix formulae use the summation only when there is no Transpose . $\endgroup$ – Bharathi Jun 28 at 17:09
  • $\begingroup$ Okay ! I am editing my question with further details $\endgroup$ – Bharathi Jun 28 at 17:24
  • $\begingroup$ @Yahya , Can you please help me understand why the above two formulae differ ? I'm sorry if I'm asking silly questions . I'm just a beginner and want to get myself clear. $\endgroup$ – Bharathi Jun 28 at 17:29

Assuming zero-mean samples, both formulations give covariance estimates. I assume $m$ denoted in the upper-most formula is either $n$ or $n-1$. The first one uses individual samples, $x_i$ and the second one uses the data matrix, $X$, where the samples are the rows. The typical formulation of the data matrix $X$ is as follows:

$$X=\begin{bmatrix}x_1^T\\ x_2^T\\\vdots\\x_n^T\end{bmatrix}$$

So, the covariance estimate $\frac{1}{n-1}X^TX$ is: $$\frac{1}{n-1}\begin{bmatrix}x_1 &x_2&\dots &x_n\end{bmatrix}\begin{bmatrix}x_1^T\\ x_2^T\\\vdots\\x_n^T\end{bmatrix}=\frac{1}{n-1}\sum_{i=1}^n x_ix_i^T$$

| cite | improve this answer | |
  • $\begingroup$ Brilliant Answer ! Thanks . But Why should data matrix X have entries as xi^T . Each xi might just be a single number right ? Transposing and not transposing a single number wouldn't make any difference , Am I right ? Or , please tell me if there is any specific reason to do so . $\endgroup$ – Bharathi Jun 28 at 19:25
  • 1
    $\begingroup$ $x_i$ is in general a vector, say $p\times 1$, where $p$ is the dimension of your data. $\endgroup$ – gunes Jun 28 at 19:26
  • 1
    $\begingroup$ Oh yeah alright ! Got it . Thanks a lot $\endgroup$ – Bharathi Jun 28 at 19:27

Your confusion seems to be about the notation being used here. You seem to be assuming that $x^{(i)}$ is a matrix. Whereas, in fact, $x^{(i)}$ is a column vector: $$ x^{(i)}= \begin{bmatrix} x_1^{(i)} \\ x_2^{(i)} \\ \vdots \\ x_p^{(i)} \end{bmatrix} $$ Thus, every $x^{(i)}$ contains a single observation of all the $p$ variables that you're computing the covariance of, where each row is a different variable. The index $i$ denotes observations, not variables. For instance, $x^{(5)}$ contains the values of all of our variables in the 5-th observation. And $x_3^{(5)}$ would be the value of the third variable in the fifth observation.

If you wanted to do it all with matrix notation, and remove the summation operator from the formula, you could define a data matrix $X$: $$ X= \begin{bmatrix} x^{(1)} & x^{(2)} & \cdots & x^{(n)} \end{bmatrix} $$ $$ = \begin{bmatrix} x_1^{(1)} & x_1^{(2)} & \cdots & x_1^{(n)} \\ x_2^{(1)} & x_2^{(2)} & \cdots & x_2^{(n)} \\ \vdots &\vdots & \ddots\ & \vdots \\ x_p^{(1)} & x_p^{(2)} & \cdots & x_p^{(n)} \end{bmatrix} $$ And then you can use the formula: $$ \Sigma = \frac{1}{m}{XX}^T $$ where $m$ equals either $n$ or $n-1$. This formula is equivalent to the one used in Andrew Ng's notes. We got rid of the summation over observations by putting those observations into the columns of the matrix $X$, and then taking a matrix product that "sums away" the observation dimension "under the hood". Importantly, though, both formulas define exactly the same sequence of operations.

In Ng's version, you can think of each $x^{(i)}{x^{(i)}}^T$-term as the "instantaneous covariance" of your variables, for the $i$-th observation (i.e. how much they happened to co-vary in that particular instance). We then average over all these instantaneous covariances to get an estimate of the overall covariance (how much the variables co-vary on average).

| cite | improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.