# Covariance Matrix

Please excuse my ignorance here. It has been awhile since I have studied linear algebra. As far as I know, a covariance matrix is often written as

COV = $\frac{1}{n-1}$ A $\cdot$ A$^T$.

Is this the same thing as

COV = $\frac{1}{n-1}$ A$^T$ $\cdot$ A ?

The reason I ask, is this can be the difference between a massive matrix or a very small matrix (for Casorati style matrices).

Ultimately, I am studying PCA, which uses the covariance matrix.

• You already answered your question: since the matrices $AA^\prime$ and $A^\prime A$ have different sizes, they cannot possibly be the same. Perhaps what you want to ask should concern what PCA-related information could be extracted from the smaller of those two matrices. – whuber Aug 17 '17 at 21:22

## 1 Answer

Edit: Whuber answered it better: $AA^T$ and $A^TA$ have different dimensions, so they won't be equal

Your question basically asks is $A A^T = A^T A$? In general, the answer is no.

Define $C=AB$ for matrices $A,B$. The element at row $i$, column $j$ of $C$ is $C_{i,j}=\sum_k A_{i,k}B_{k,j}$, by the definition of matrix multiplication. If $AA^T=A^TA$, then element-wise they should be the same. Of course, A must have the same number of rows and columns, otherwise the two products have different dimensions, so the answer is no in that case. If $A$ is a square matrix, though, we can write the condition needed for the equality to hold:

If we let $C=AA^T$, then $C_{i,j}=\sum_k A_{i,k} (A^T)_{k,j}=\sum_k A_{i,k} A_{j,k}$ by the property of matrix transpose $A_{i,j} = (A^T)_{j,i}$.

Now, consider $C'=A^TA$. We have that $C'_{i,j}=\sum_k (A^T)_{i,k} A_{k,j}=\sum_k A_{k,i} A_{k,j}$.

Since, in general, $\sum_k A_{k,i} A_{k,j} \neq \sum_k A_{i,k} A_{j,k}$, we have that $A A^T \neq A^T A$ for $A$ a square matrix.

Some notable exceptions for which $A^TA=AA^T$ are the identity matrix and symmetric matrices where $A=A^T$, but since in PCA $A$ is usually a design matrix (rows are observations, columns are features), you won't be any of these exceptions.

• Sweet name btw. Essentially, I am generating eigenfaces. What I do, is calculated the $COV$ using $A^TA$, as the matrix is tiny (20x20). I then project my Casorati matrix (20,000x20) of face images onto the small $COV$ matrix. This yields 20 images, of principal components. So perhaps I am confused why I can do this? – Shinobii Aug 17 '17 at 21:03
• Given a design matrix $X$ with $N$ rows and $M$ features, so $X\in \mathbb R^N \times \mathbb R^M$, the covariance matrix $XX^T$ will have elements $C_{i,J} = cov(x_i, x_j)$, where $x_i, x_j$ are your features. The covariance is estimated using all of the observations you have (all $N$ for each variable), so thats how it collapses down to a 20x20 matrix in your case (because you have 20 features, it seems). What do you mean when you say project the 20,000x20 matrix onto the covariance matrix? Did you diagonalize the covariance matrix to get the principal components and just change the basis? – the higgs broson Aug 17 '17 at 21:08
• Sorry, my math lingo is terrible. Also, I missed a step. I calculate the eigenvectors ($E$) of the 20x20 $COV$ matrix (yields 20x20 $E$ matrix). Then I take my demeaned images ($DM$) and multiply them to the $E$ matrix ($DM \times E$), which is 20,000x20 $\times$ 20x20 $\rightarrow$ my 20 images. I hope this is clear. – Shinobii Aug 17 '17 at 21:12
• Perhaps I'm misunderstanding, but just multiplying $DM \times E$ doesn't do a change of basis. If you have a matrix $A$ and your matrix of eigenvectors $E$, to change the basis of $A$ you need to do $A'=E^{-1} A E$. Also, just to confirm: you only have 20 features? Whether or not the covariance matrix is given by $AA^T$ vs $A^TA$ depends on whether the rows of $A$ are observations and the columns are features, or visa versa. For the former (which is more standard), $AA^T$ is the covariance matrix, and for the later, you need to do $A^TA$. Which one are you using? – the higgs broson Aug 17 '17 at 21:17
• A matrix $A$ for which $A^\prime A = AA^\prime$ is called a normal matrix. Such matrices form the conjugacy class of the diagonal matrices: that is, the Spectral Theorem applies to them. The argument concerning why not all square matrices are normal might be made more clear by providing a simple counterexample, such as this one:$$\pmatrix{0&1\\0&0}^\prime\pmatrix{0&1\\0&0}=\pmatrix{0&0\\0&1}\ne\pmatrix{1&0\\0&0}=\pmatrix{0&1\\0&0}\pmatrix{0&1\\0&0}^\prime.$$ – whuber Aug 17 '17 at 21:49