1
$\begingroup$

The "transpose trick" in principal component analysis (PCA) involves replacing the computation of $\mathrm{eig}(A^TA)$ by $\mathrm{eig}(AA^T)$, as it is known that if $v$ is an eigenvector of $AA^T$, then $A^Tv$ is an eigenvector of $A^TA$ - compare https://stats.stackexchange.com/a/7144/77888.

However, I fail to see under which conditions that trick actually applies to PCA, as PCA "usually" involves "a normalization step of the initial data" that "consists of mean centering" (Wikipedia).

The Stats link above simply says that

for simplicity, suppose that the columns of A have already been normalized to have zero mean,

while Fundamentals of Statistics says nothing about that whatsoever - despite explicitly stating that

the PCA is based on the solution of the eigenvalue problem for the covariance matrix

and stating that in the covariance matrix,

the mean of each variable is subtracted before multiplication.

So, what conditions need to be fulfilled for the transpose trick to work be directly applicable in PCA of a generally non-normalized matrix? That is, when is $\mathrm{eig}(\mathrm{cov}(A))$ related to $\mathrm{eig}(\mathrm{cov}(A^T))$?

My motivation for this question stems from the fact that I have seen someone use [v,~]=eig(cov(data')); v(:,end) (instead of [v,~]=eig(cov(data)); data * v(:, end)) in MATLAB to project the data onto the principal component associated with the largest eigenvalue, and I have the strong feeling that this substitution is valid (meaning, I get the same result up to some factor) only if the data has zero-mean rows and columns [because both data and data' are used in cov].

$\endgroup$
  • 1
    $\begingroup$ I don't understand why there is even a question to ask here, because the "transpose trick" is applied to whatever matrix $A$ you're analyzing. If you're analyzing a matrix that has been normalized, then that's what "$A$" refers to. If you don't normalize, then $A$ refers to the original matrix. The real problem with this trick is that by squaring the matrix it halves the number of significant digits that can be computed, which leads to numerical instabilities when $A$ is ill-conditioned. $\endgroup$ – whuber Feb 12 at 14:25
  • $\begingroup$ @whuber I agree that my original question may have been unclear in the exact question (but I would have expected that the context of that question had made it clear). Anyway, I have clarified the question: If my matrix A is not normalized, does the transpose trick (which involves $eig$, mostly) carry over to PCA of the matrix and its transpose? And if not, when does it? $\endgroup$ – bers Feb 13 at 15:15
  • $\begingroup$ It remains unclear what you're trying to ask. The question you present explicitly is "when is eig(cov(A)) related to eig(cov(A'))" and you provide the answer to that in the very beginning. Since it's purely a theorem about matrices, and no matter how you do PCA you will be doing it with a matrix, why is there any question about its applicability? $\endgroup$ – whuber Feb 13 at 17:19
  • $\begingroup$ @whuber user20160 was perfectly able to understand my question :) $\endgroup$ – bers Feb 14 at 14:02
  • $\begingroup$ That means one person out of dozens who have viewed it interpreted your post as intended--but it provides only slim evidence that the post is clear. $\endgroup$ – whuber Feb 14 at 15:35
2
$\begingroup$

The 'transpose trick' is a general fact of linear algebra, so it always holds. But, transposing the data matrix before estimating the covariance matrix (as in the Matlab code you quoted) is not the proper way to use this fact to do PCA. This has to do with the centering issues you mentioned.

What goes wrong

Suppose the data matrix is $A$, with $n$ observations on the rows and $p$ features on the columns. Let $A_c$ denote the centered data matrix, obtained by centering the columns of $A$. The sample covariance matrix is

$$C = \frac{1}{n} A_c^T A_c$$

But, if you transpose $A$ before computing the sample covariance matrix, you end up with:

$$C_* = \frac{1}{p} A_{c*} A_{c*}^T$$

where $A_{c*}$ is the result of centering the rows of $A$. Notice that $C$ and $C_*$ don't have the relationship needed to use the transpose trick, unless $A_c = A_{c*}$ (as happens when the row and column means are both zero, for example). And, in the latter case, the eigenvalues of $C_*$ will still differ from those of $C$ by a constant factor of $\frac{n}{p}$.

The following procedure should be used instead.

How to do PCA using the transpose trick

Given $n \times p$ data matrix $A$ (with $p > n$, otherwise this is just wasted computation):

  1. Center the columns of $A$ to obtain the centered data matrix $A_c$.

  2. Let $G = \frac{1}{n} A_c A_c^T$. Note that $G$ and $C$ (above) do have the proper relationship to use the transpose trick.

  3. Let $V \Lambda V^T$ be the eigendecomposition of $G$ (with eigenvectors on the columns of $V$ and eigenvalues on the diagonal of $\Lambda$).

  4. Let $\tilde{V} = A_c^T V$

The diagonal of $\Lambda$ and the columns of $\tilde{V}$ are equal to the top $n$ eigenvalues and eigenvectors of the sample covariance matrix $C$. Any remaining eigenvalues of $C$ are zero. It's a good idea to scale the columns of $\tilde{V}$ to have unit norm, as they won't generally be normalized to begin with.

$\endgroup$
  • $\begingroup$ This is exactly what I was asking - thank you! $\endgroup$ – bers Feb 14 at 14:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.