Why are there only $n-1$ principal components for $n$ data if the number of dimensions is $\ge n$? - Cross Validated most recent 30 from stats.stackexchange.com 2019-09-16T16:26:01Z https://stats.stackexchange.com/feeds/question/123318 https://creativecommons.org/licenses/by-sa/4.0/rdf https://stats.stackexchange.com/q/123318 22 Why are there only $n-1$ principal components for $n$ data if the number of dimensions is $\ge n$? GrokingPCA https://stats.stackexchange.com/users/60334 2014-11-09T19:22:35Z 2017-10-19T15:13:54Z <p>In PCA, when the number of dimensions $d$ is greater than (or even equal to) the number of samples $N$, why is it that you will have at most $N-1$ non-zero eigenvectors? In other words, the rank of the covariance matrix amongst the $d\ge N$ dimensions is $N-1$.</p> <p><strong>Example:</strong> Your samples are vectorized images, which are of dimension $d = 640\times480 = 307\,200$, but you only have $N=10$ images.</p> https://stats.stackexchange.com/questions/123318/why-are-there-only-n-1-principal-components-for-n-data-if-the-number-of-dime/123349#123349 20 Answer by gung for Why are there only $n-1$ principal components for $n$ data if the number of dimensions is $\ge n$? gung https://stats.stackexchange.com/users/7290 2014-11-10T03:56:00Z 2017-01-30T17:47:03Z <p>Consider what PCA does. Put simply, PCA (as most typically run) creates a new coordinate system by: </p> <ol> <li>shifting the origin to the centroid of your data, </li> <li>squeezes and/or stretches the axes to make them equal in length, and </li> <li>rotates your axes into a new orientation. </li> </ol> <p>(For more details, see this excellent CV thread: <a href="https://stats.stackexchange.com/q/2691/7290">Making sense of principal component analysis, eigenvectors &amp; eigenvalues</a>.) However, it doesn't just rotate your axes any old way. Your new $X_1$ (the first principal component) is oriented in your data's direction of maximal variation. The second principal component is oriented in the direction of the next greatest amount of variation <em>that is orthogonal to the first principal component</em>. The remaining principal components are formed likewise. </p> <p>With this in mind, let's examine <a href="https://stats.stackexchange.com/questions/123318/why-are-there-only-n-1-principal-axes-for-n-data-points-if-the-number-of-dim/123349#comment235087_123318">@amoeba's example</a>. Here is a data matrix with two points in a three dimensional space:<br> $$X = \bigg[ \begin{array}{ccc} 1 &amp;1 &amp;1 \\ 2 &amp;2 &amp;2 \end{array} \bigg]$$ Let's view these points in a (pseudo) three dimensional scatterplot: </p> <p><img src="https://i.stack.imgur.com/IST5N.png" alt="enter image description here"></p> <p>So let's follow the steps listed above. (1) The origin of the new coordinate system will be located at $(1.5, 1.5, 1.5)$. (2) The axes are already equal. (3) The first principal component will go diagonally from $(0,0,0)$ to $(3,3,3)$, which is the direction of greatest variation for these data. Now, the second principal component must be orthogonal to the first, and should go in the direction of the greatest <em>remaining</em> variation. But what direction is that? Is it from $(0,0,3)$ to $(3,3,0)$, or from $(0,3,0)$ to $(3,0,3)$, or something else? <em>There is no remaining variation, so there cannot be any more principal components</em>. </p> <p>With $N=2$ data, we can fit (at most) $N-1 = 1$ principal components. </p>