6
$\begingroup$

Let $A$ a be an $n$ x $p$ matrix and let $B$ be the transformed data set of $A$ under $Q$:

$$ B = A Q $$

where Q is a $p$ x $p$ orthogonal matrix:

$$ Q Q^T = I $$

$n$ is the number of samples (rows) and $p$ is the number of features (columns).

 

$A$ and $B$ are centred (zero mean for each feature).

Now suppose we perform PCA on both $A$ and $B$:

$$ \frac{1}{n-1} A^T A = V_A L_A V^T_A $$

$$ \frac{1}{n-1} B^T B = V_B L_B V^T_B $$

The question is, are the projections of $A$ and $B$ on their principal directions equal and why?

$$ A V_A = B V_B \ ? $$

That is, are the datasets identical upon "transforming" them with PCA?


Later edit: I simplified the question to only refer to orthogonal transformations (rotations). The original question referred to translations as well (the explanation in that case is trivial).

$\endgroup$
  • 1
    $\begingroup$ You may want to change "Suppose $A$ and $B$ are two $n x p$ matrices representing two data sets" to something along the lines: "let $A$ a be an $n x p$ matrices and let $B$ be the transformed data set of $A$ under $Q$." just for clarity $\endgroup$ – Joseph Santarcangelo Oct 15 '16 at 16:45
  • $\begingroup$ @JosephSantarcangelo thanks, I've fixed that. Feel free to adjust the question if it isn't clear enough. $\endgroup$ – turdus-merula Oct 15 '16 at 17:02
3
$\begingroup$

The covariance matrix of $B$ is decomposed as:

$$ \frac{1}{n-1} B^T B = V_B L_B V^T_B $$

If we rewrite the covariance matrix of $B$ in terms of $A$ and $Q$ we have:

$$ \frac{1}{n-1} B^T B = \frac{1}{n-1} (AQ)^T AQ = \frac{1}{n-1} Q^T A^T A Q = Q^T (\frac{1}{n-1} A^T A) Q = Q^T V_A L_A V^T_A Q $$

We observe that the covariance matrices of $A$ and $B$ are generally different:

$$ V_A L_A V^T_A \neq Q^T V_A L_A V^T_A Q $$

We identify (wasn't sure about this step at first):

$$ V_B = Q^T V_A $$ $$ L_B = L_A $$ $$ V^T_B = (Q^T V_A)^T = V^T_A Q $$

And now we have:

$$ B V_B = A Q Q^T V_A = A V_A $$

Which means that the datasets are identical upon "transforming" them with PCA.


Many thanks to @amoeba and @Alex R.

$\endgroup$
  • 1
    $\begingroup$ +1, this is correct. The step you are not sure about is legit because the eigen-decomposition is unique (assuming that all eigenvalues are distinct). So if you know that $Q^\top V_A$ is an orthogonal matrix and that $L_B$ is a diagonal matrix, then you know that $Q^\top V_AL_AV^\top_AQ$ is the eigen-decomposition. $\endgroup$ – amoeba says Reinstate Monica Oct 15 '16 at 21:23
1
$\begingroup$

Let $R$ be a rotation (i.e. an orthogonal matrix: $R^T=R^{-1}$). $A=U\Sigma V^*$ by SVD, and correspondingly $A^TA=V\Sigma^2 V^*$.

On the other hand if $B:=RA$, then:

$$B^TB=A^TR^TRA=A^TA=V\Sigma^2V^*.$$

So PCA is "invariant" under rotations applied to $A$ from the left. Specifically, this means that the covariance matrix is invariant. However the actual projections will differ. Specifically, $A=U\Sigma V^*$, whereas $B=RA=RU\Sigma V^*$, giving $B=W\Sigma V^*$, where $W:=R^*U$.

$\endgroup$
  • $\begingroup$ I will "accept this answer" in a few days. I would like to clean up my question a bit first. $\endgroup$ – turdus-merula Oct 10 '16 at 8:04
  • 2
    $\begingroup$ @user115202 I don't think this answer is correct. "Rotating" a data cloud will change the covariance matrix (the eigenvectors will rotate too), but the projections will stay the same. Because such a rotation corresponds to rotations applied to $A$ from the right, not from the left, assuming $A$ is $n\times p$. $\endgroup$ – amoeba says Reinstate Monica Oct 15 '16 at 0:16
  • $\begingroup$ @amoeba and Alex, I attempted an explanation. Do you think you can check it sometimes? Thank you for your answers and comments. $\endgroup$ – turdus-merula Oct 15 '16 at 17:37
  • $\begingroup$ -1. I am sorry, I feel I have to downvote this answer because in its current form it can be misleading. Please ping me if you edit, Alex, I will be happy to remove the downvote. CC to @user115202. $\endgroup$ – amoeba says Reinstate Monica Oct 15 '16 at 21:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.