16
$\begingroup$

When I run PCA on a certain data set, is the solution given to me unique?

I.e., I obtain a set of 2d coordinates, based on interpoint distances. Is it possible to find at least one more arrangement of the points that would meet these constraints?

If the answer is yes, how can I find such different solution?

$\endgroup$
3
  • 12
    $\begingroup$ The answer to the uniqueness question is both yes and no. It is "yes" in the sense that the eigenspaces and eigenvalues are mathematically well and uniquely defined. It is "no" in the senses that (a) there are multiple ways to represent those eigenspaces (even a normalized eigenvector can be negated and there are many choices of basis for degenerate eigenspaces) and (b) different algorithms may produce results that differ due to accumulation of floating point error in the calculations. $\endgroup$
    – whuber
    Oct 10, 2012 at 15:48
  • $\begingroup$ Ramsay and Silverman in the book "Functinal Data Analysis", mention VARIMAX rotation. Thy talk about splitting a dataset of functions (represented as a matrix) into its principle components. $\endgroup$
    – power
    Nov 9, 2012 at 6:38
  • $\begingroup$ It sounds like you want to use PCA as a tool for dimension reduction. You may start by looking at Dimensionality reduction... $\endgroup$
    – Elvis
    Dec 9, 2012 at 6:41

3 Answers 3

10
$\begingroup$

Something that hasn't been noticed yet is that simply reversing the sign of a PC produces a different solution. That is, if $\mathbf{w}$ is the $n$th principal component, then $-\mathbf{w}$ is also a solution to the $n$th principal component. This has caused confusion before, especially when your computer outputs alternating PCs. See this question.

$\endgroup$
1
  • 3
    $\begingroup$ For an interesting practical application of this ambiguity, please see stats.stackexchange.com/questions/34396. (BTW, the sign reversal was noticed: see the first comment to this question.) $\endgroup$
    – whuber
    Jan 8, 2013 at 16:40
9
$\begingroup$

No, the answer is not unique. There are many ways to show this. One possibility is to notice that spectral decomposition of a square $p$ by $p$ matrix $X$ is the solution to the maximization of a convex function of $w$. Consider the first eigen-vector/value:

$$\lambda_1=\underset{w\in\mathbb{R}^{p}:||w||=1}{\max} w'Xw$$

(where $\lambda_1$ is the first eigen-value and $w^*$ the first eigen-vector).

The solution to such problems (e.g. the values of $w$ attaining that maximum) are, in general, not unique.

However the algorithms for computing these solutions are deterministic, meaning that save for numerical corner cases, the solutions you get should be the same.

Example of such numerical corner cases: cases where several eigen-values are (numerically) the same, cases where the $X$ is rank-deficient...

$\endgroup$
0
$\begingroup$

It depends.

If the eigenvalues of the covariance matrix are different, then the PCA is unique. Else not.


The fact that the variances of the principal components are given by λi has an important implication for the uniqueness of PCA. If two of the eigenvalues are equal, then the variance of those principal components are equal. Then, the principal components are not well-defined anymore, because we can make a rotation of those principal components without affecting their variances. This is because if p zi and zi+1 have the same variance, then linear combinations such as 1/2zi + p 1/2zi+1 and p 1/2zi − p 1/2zi+1 have the same variance as well; all the constraints (unit variance and orthogonality) are still fulfilled, so these are equally valid principal components. In fact, in linear algebra, it is well-known that the eigenvalue decomposition is uniquely defined only when the eigenvalues are all distinct.

Source: Principal component analysis; Aapo Hyvärinen; Based on material from the book Natural Image Statistics,2009; https://www.mv.helsinki.fi/home/amoaning/movies/uml/pca_handout.pdf

Or

PCA is unique up to signs, if the eigenvalues of the covariance matrix are different from each other.

Is PCA unique or not, that is, is there only one PCA solution. Multiple solutions may fulfill the PCA criteria. We consider the decomposition X = Y UT where U is orthogonal, Y T Y = Dm with Dm as m-dimensional diagonal matrix, and the eigenvalues of Dm are sorted increasingly.

Source: Machine Learning: Unsupervised Techniques; 2014; Sepp Hochreiter; Institute of Bioinformatics Johannes Kepler University Linz

$\endgroup$
2
  • $\begingroup$ It's challenging to read this answer because it does not use math typesetting. For information on how to use typesetting, see math.meta.stackexchange.com/questions/5020/… $\endgroup$
    – Sycorax
    May 3, 2021 at 16:30
  • $\begingroup$ As a factual matter, the sentence " PCA is unique up to signs, if the eigenvalues of the covariance matrix are different from each other," seems to be in conflict with the claim "If the eigenvalues of the covariance matrix are different, then the PCA is unique. Else not." because we can imagine a covariance matrix $C$ with all eigenvalues unique and positive. However, if $v$ is an eigenvector to $C$, then $-v$ is also an eigenvector to $C$. $\endgroup$
    – Sycorax
    May 3, 2021 at 16:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.