3
$\begingroup$

The rotation matrix outputted by the PCA algorithm should be independent of something trivial like the column ordering of the source data. Can anyone explain why my output diverges from my expectation for consistency?

I made a test input file 30x569 from a pre-made dataset

from sklearn.datasets import load_breast_cancer
cancer = load_breast_cancer()
cancer.keys()

df = pd.DataFrame(cancer['data'],columns=cancer['feature_names'])
df.to_csv(r'input file',index=False)

Then generated a 30x30 output with all the covariance-based PCA components

import pandas as pd
import numpy as np

daily_series = pd.read_csv (r'input path')

sd = daily_series[daily_series.columns[0:daily_series.shape[1]]]
scaled_data = sd #unscaled
from sklearn.decomposition import PCA
pca = PCA(n_components=daily_series.shape[1])
pca_model = pca.fit(scaled_data)
components = ['PC1','PC2','PC3','PC4','PC5','PC6','PC7','PC8','PC9','PC10','PC11','PC12','PC13','PC14','PC15','PC16','PC17','PC18','PC19','PC20','PC21','PC22','PC23','PC24','PC25','PC26','PC27','PC28','PC29','PC30']
variables = daily_series.columns[0:daily_series.shape[1]]
Matrix = pd.DataFrame(pca_model.components_, columns=components, index=variables)

Matrix.to_csv(r'output path', index=True)

When I reorder the columns (let's say alphabetically) of the test input file. And run the above my output from the original test is different not just in the signs but also magnitude. I don't understand how that's possible.

Output (left is original output/right is output after alphabetizing columns in source data): enter image description here

$\endgroup$
8
  • 1
    $\begingroup$ The main thing to observe is that the SVD of $A$ is given by $$ A = U\Sigma V^\top $$ then for a column permutation via matrix $P$ we have $$ AP = U\Sigma V^\top P $$ because permutation matrices are orthogonal and products of orthogonal matrices are orthogonal. Your components are given by $V^\top P$. The rest is just showing the relationship between PCA and SVD. This is covered thoroughly in stats.stackexchange.com/questions/134282/… $\endgroup$
    – Sycorax
    Jan 30, 2020 at 0:27
  • $\begingroup$ @SycoraxsaysReinstateMonica The transformed variables (principal components) should correspond to the original variables. So why don't they as I highlighted in red? So I'm trying to understand why the python PCA implementation doesn't work as it should. I shouldn't have to write my own implementation right? $\endgroup$
    – Quesop
    Jan 30, 2020 at 4:36
  • $\begingroup$ The transformed variables do correspond to the original variables. The reason the two outputs don't match is that one is a permutation of the other. This is what I wrote in my first comment. The python implementation works correctly because this is a property of SVD and orthogonal matrices. $\endgroup$
    – Sycorax
    Jan 30, 2020 at 4:41
  • $\begingroup$ @SycoraxsaysReinstateMonica I know what a permutation is and I know what column is but I don't understand what you mean by "column permutation." Permutation means the order matters. Manifestly the order of the columns matters but why? Column order is trivial so it should not matter. How are the left and the right screenshots equivalent representations? $\endgroup$
    – Quesop
    Jan 30, 2020 at 4:47
  • 1
    $\begingroup$ @SycoraxsaysReinstateMonica I looked at the output files again and I see you were right. PC1, PC2, PC3 became PC12, PC15, PC11 respectively when I alphabetized columns in my example. Ok this means the relationship between variables after you reduce dimensions is sensitive to the ordering of the columns in the beginning. How do you pick the "best" order of columns in the beginning if your goal is to understand how the variables are interrelated? $\endgroup$
    – Quesop
    Jan 30, 2020 at 6:12

1 Answer 1

4
$\begingroup$

We have thoroughly developed the relationship between SVD and PCA in Relationship between SVD and PCA. How to use SVD to perform PCA? which is worth reviewing if you're uncertain about the connection.


The sklearn PCA implementation is working correctly.

The main thing to observe is that the SVD of $A$ is given by $$ A=U S V^\top $$ so for a permutation of columns via matrix $P$ we have $$ AP=U S V^\top P. $$

Another way to state this is that if you compute the SVD of $AP$, you'll end up with $AP = U S \tilde{V}^\top$, where $\tilde{V}^\top = V^\top P$.

We know that $\tilde{V}^\top=V^\top P$ is orthogonal because permutation matrices are orthogonal and products of orthogonal matrices are orthogonal.

Your screenshots show different things because you're comparing $V^\top$ and $V^\top P$, which are not equal in general. In fact, $V^\top$ and $V^\top P$ are only guaranteed to be equal if $P=I$. Column order matters just for $V$; $U$ and $S$ are the same.


We can even show that a permutation yields the same orthogonal rotation.

$$ \begin{aligned} AV &= USV^\top V \\ AV &= US \end{aligned} $$

And we can show the same result for $AP$ because a permutation matrix $P$ is orthogonal.

$$ \begin{aligned} AP P^\top V &= USV^\top P P^\top V \\ AV &= US \end{aligned} $$

In other words, the column order doesn't matter for creating a linearly independent basis for $A$, because you obtain the same result for $AP$ and $A$.


We can demonstrate this all in Python.

import numpy as np
from numpy.linalg import svd
from numpy.random import shuffle
from sklearn.datasets import load_breast_cancer

if __name__ == "__main__":
  X, y = load_breast_cancer(True)
  U, S, V = svd(X, full_matrices=False)

  P = np.eye(X.shape[1])
  shuffle(P)

  print("X and X @ P are not the same.")
  print(X @ P - X)

  # This will work correctly because both X and the SVD of X are permuted.
  assert np.allclose(U @ np.diag(S) @ V @ P - X @ P, 0.0)

  try:
    # This will fail because X is permuted but the SVD is ~not~.
    assert np.allclose(U @ np.diag(S) @ V - X @ P, 0.0)
  except AssertionError:
    print("V @ P != V")
    print(V @ P - V)

You can replace P with any permutation matrix you desire, even one which alphabetizes the column names.

$\endgroup$
18
  • $\begingroup$ Actually I am still confused by this Q. Shouldn't PCA from sklearn order components by explained variance? If so, permuting the features in X should not affect the returned V, because its columns are supposed to be sorted. What am I missing? $\endgroup$
    – amoeba
    Feb 4, 2020 at 8:54
  • $\begingroup$ They are ordered by explained variance. The $U$ and $S$ in my answer are the same for both $AP$ and $A$. The difference is that the $P$ re-arranges which directions maximize variance, kind of like how re-labeling "North" on a compass doesn't change where the North Pole is. Another way to think about it is that the covariance of $AP$ is given by $\frac{1}{n-1} P^\top A^\top AP = P^\top V\frac{S^2}{n-1} V^\top P$, which is a full permutation of $A$'s covariance $V \frac{S^2}{n-1} V^\top$ . Clearly the factors $S, V$ are just re-arranged by $P$, but its values aren't changed. $\endgroup$
    – Sycorax
    Feb 4, 2020 at 14:21
  • $\begingroup$ Another way to answer your question is to point out that it's just convention to sort an SVD factorization according to the values of $S$. However, the definition of a singular value and singular vector (and likewise eigenvalue and eigenvector) is still satisfied as long as you match singular values in $S$ to their singular vectors in $U,V$. So for whatever ordering of $S$ you choose, all of the equalities in my answer will still hold, as long as you order $U,V$ to match the ordering in $S$. $\endgroup$
    – Sycorax
    Feb 4, 2020 at 14:27
  • $\begingroup$ Sorry but this does not clarify :-( Maybe I am misunderstanding what happens in the question? The breast cancer dataset has 30 features and 500+ samples. OP does PCA on 30x30 cov matrix using sklearn and gets 30 eigenvectors. They should be sorted by the eigenvalues. Now the OP permutes the 30 features and re-runs PCA. The eigenvectors are not the same even accounting for the re-ordering: e.g. the element for "area error" in the 1st eigenvector is 0.08 vs. 0.05. In the comments OP says that PC1 became PC12. This does not make sense to me. $\endgroup$
    – amoeba
    Feb 4, 2020 at 22:41
  • 1
    $\begingroup$ @amoeba I see what you're saying, but I've found that reliance on terminology can confuse the issue, especially because most software documentation is garbage. Instead, I prefer to think through the factorization. So in your example, if I want the product $AV$ to have rank $k$, I just think through the algebra: what shapes of $U,S,V$ do I need for the (truncated) SVD to make sense and produce the desired result? Maybe it's just a personal quirk, but this is the only way I can remember how PCA and SVD work. What's a "score," "loading," or "PC"? I honestly can never remember, but I know the math $\endgroup$
    – Sycorax
    Feb 5, 2020 at 23:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.