# Is PCA invariant to orthogonal transformations?

Let $A$ a be an $n$ x $p$ matrix and let $B$ be the transformed data set of $A$ under $Q$:

$$B = A Q$$

where Q is a $p$ x $p$ orthogonal matrix:

$$Q Q^T = I$$

$n$ is the number of samples (rows) and $p$ is the number of features (columns).

$A$ and $B$ are centred (zero mean for each feature).

Now suppose we perform PCA on both $A$ and $B$:

$$\frac{1}{n-1} A^T A = V_A L_A V^T_A$$

$$\frac{1}{n-1} B^T B = V_B L_B V^T_B$$

The question is, are the projections of $A$ and $B$ on their principal directions equal and why?

$$A V_A = B V_B \ ?$$

That is, are the datasets identical upon "transforming" them with PCA?

Later edit: I simplified the question to only refer to orthogonal transformations (rotations). The original question referred to translations as well (the explanation in that case is trivial).

• You may want to change "Suppose $A$ and $B$ are two $n x p$ matrices representing two data sets" to something along the lines: "let $A$ a be an $n x p$ matrices and let $B$ be the transformed data set of $A$ under $Q$." just for clarity – Joseph Santarcangelo Oct 15 '16 at 16:45
• @JosephSantarcangelo thanks, I've fixed that. Feel free to adjust the question if it isn't clear enough. – turdus-merula Oct 15 '16 at 17:02

The covariance matrix of $B$ is decomposed as:

$$\frac{1}{n-1} B^T B = V_B L_B V^T_B$$

If we rewrite the covariance matrix of $B$ in terms of $A$ and $Q$ we have:

$$\frac{1}{n-1} B^T B = \frac{1}{n-1} (AQ)^T AQ = \frac{1}{n-1} Q^T A^T A Q = Q^T (\frac{1}{n-1} A^T A) Q = Q^T V_A L_A V^T_A Q$$

We observe that the covariance matrices of $A$ and $B$ are generally different:

$$V_A L_A V^T_A \neq Q^T V_A L_A V^T_A Q$$

$$V_B = Q^T V_A$$ $$L_B = L_A$$ $$V^T_B = (Q^T V_A)^T = V^T_A Q$$

And now we have:

$$B V_B = A Q Q^T V_A = A V_A$$

Which means that the datasets are identical upon "transforming" them with PCA.

Many thanks to @amoeba and @Alex R.

• +1, this is correct. The step you are not sure about is legit because the eigen-decomposition is unique (assuming that all eigenvalues are distinct). So if you know that $Q^\top V_A$ is an orthogonal matrix and that $L_B$ is a diagonal matrix, then you know that $Q^\top V_AL_AV^\top_AQ$ is the eigen-decomposition. – amoeba says Reinstate Monica Oct 15 '16 at 21:23

Let $R$ be a rotation (i.e. an orthogonal matrix: $R^T=R^{-1}$). $A=U\Sigma V^*$ by SVD, and correspondingly $A^TA=V\Sigma^2 V^*$.

On the other hand if $B:=RA$, then:

$$B^TB=A^TR^TRA=A^TA=V\Sigma^2V^*.$$

So PCA is "invariant" under rotations applied to $A$ from the left. Specifically, this means that the covariance matrix is invariant. However the actual projections will differ. Specifically, $A=U\Sigma V^*$, whereas $B=RA=RU\Sigma V^*$, giving $B=W\Sigma V^*$, where $W:=R^*U$.

• I will "accept this answer" in a few days. I would like to clean up my question a bit first. – turdus-merula Oct 10 '16 at 8:04
• @user115202 I don't think this answer is correct. "Rotating" a data cloud will change the covariance matrix (the eigenvectors will rotate too), but the projections will stay the same. Because such a rotation corresponds to rotations applied to $A$ from the right, not from the left, assuming $A$ is $n\times p$. – amoeba says Reinstate Monica Oct 15 '16 at 0:16
• @amoeba and Alex, I attempted an explanation. Do you think you can check it sometimes? Thank you for your answers and comments. – turdus-merula Oct 15 '16 at 17:37
• -1. I am sorry, I feel I have to downvote this answer because in its current form it can be misleading. Please ping me if you edit, Alex, I will be happy to remove the downvote. CC to @user115202. – amoeba says Reinstate Monica Oct 15 '16 at 21:22