When I run PCA on a certain data set, is the solution given to me unique?

I.e., I obtain a set of 2d coordinates, based on interpoint distances. Is it possible to find at least one more arrangement of the points that would meet these constraints?

If the answer is yes, how can I find such different solution?

  • 11
    $\begingroup$ The answer to the uniqueness question is both yes and no. It is "yes" in the sense that the eigenspaces and eigenvalues are mathematically well and uniquely defined. It is "no" in the senses that (a) there are multiple ways to represent those eigenspaces (even a normalized eigenvector can be negated and there are many choices of basis for degenerate eigenspaces) and (b) different algorithms may produce results that differ due to accumulation of floating point error in the calculations. $\endgroup$ – whuber Oct 10 '12 at 15:48
  • $\begingroup$ Ramsay and Silverman in the book "Functinal Data Analysis", mention VARIMAX rotation. Thy talk about splitting a dataset of functions (represented as a matrix) into its principle components. $\endgroup$ – power Nov 9 '12 at 6:38
  • $\begingroup$ It sounds like you want to use PCA as a tool for dimension reduction. You may start by looking at Dimensionality reduction... $\endgroup$ – Elvis Dec 9 '12 at 6:41

Something that hasn't been noticed yet is that simply reversing the sign of a PC produces a different solution. That is, if $\mathbf{w}$ is the $n$th principal component, then $-\mathbf{w}$ is also a solution to the $n$th principal component. This has caused confusion before, especially when your computer outputs alternating PCs. See this question.

  • 3
    $\begingroup$ For an interesting practical application of this ambiguity, please see stats.stackexchange.com/questions/34396. (BTW, the sign reversal was noticed: see the first comment to this question.) $\endgroup$ – whuber Jan 8 '13 at 16:40

No, the answer is not unique. There are many ways to show this. One possibility is to notice that spectral decomposition of a square $p$ by $p$ matrix $X$ is the solution to the maximization of a convex function of $w$. Consider the first eigen-vector/value:

$$\lambda_1=\underset{w\in\mathbb{R}^{p}:||w||=1}{\max} w'Xw$$

(where $\lambda_1$ is the first eigen-value and $w^*$ the first eigen-vector).

The solution to such problems (e.g. the values of $w$ attaining that maximum) are, in general, not unique.

However the algorithms for computing these solutions are deterministic, meaning that save for numerical corner cases, the solutions you get should be the same.

Example of such numerical corner cases: cases where several eigen-values are (numerically) the same, cases where the $X$ is rank-deficient...


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.