# PCA: Iteratively reducing dimensions

ist there a difference between Iteratively removing one dimension with the least variance by computing pca and repeating this process until the data has k dimensions less...

Compared to calculating pca and removing the k dimensions with the least variance at once.

Thank you!

I believe there are shorter explanations, but here is my bit on it: PCA component axes are eigenvectors of the data covariance, and if $$u$$ is a normalized eigenvector, $$-u$$ is also one. Since you can choose one of them arbitrarily, iterative PCA does not guarantee the same output. However, if we put a rule on the eigenvectors such that they're uniquely determined, e.g. the first non-zero element will be positive, the two procedures will produce same results.

Without loss of generality, we can assume that our data is centered, i.e. has zero mean; let's call it as $$X_{n\times k}$$, where $$n$$ is the number of samples and $$k$$ is the number of dimensions. Let the eigen decomposition of $$X^TX$$ be $$US_kU^T$$, where the diagonal elements of $$S_k$$ are in non-increasing order. The first $$k-1$$ columns of $$U$$ will define the first $$k-1$$ component axes. When we omit the last column, i.e. the column with the least explained variance, the new data is $$X_1=X\ [u_1\ldots u_{k-1}]$$, which also has zero mean.

Now, we apply PCA again on this one: \begin{align}X_1^TX_1&=\begin{bmatrix} u_1^T \\ \vdots \\u_{k-1}^T \end{bmatrix}X^TX\begin{bmatrix}u_1 \ldots u_{k-1}\end{bmatrix}=\begin{bmatrix} u_1^T \\ \vdots \\u_{k-1}^T \end{bmatrix}US_kU^T\begin{bmatrix}u_1 \ldots u_{k-1}\end{bmatrix}=\underbrace{S_{k-1}}_{U_1S_{k-1}U_1^T,\ \ U_1=I}\end{align}

$$S_{k-1}$$ denotes the upper-left $$k-1\times k-1$$ part of $$S_k$$, containing the largest $$k-1$$ eigenvalues. So, the new data scatter matrix is already in diagonal form, which means the eigenvectors are canonical vectors, i.e. $$U_1=I_{k-1}$$. When we do one more iteration, we calculate the following: $$X_2=X_1[e_1\ldots e_{k-2}]=[x_{1,1}\ldots x_{1,k-1}]$$ where $$x_{1,j}$$ denotes the $$j$$-th column of $$X_1$$, and $$e_i$$ is the canonical vector, i.e. $$e_i=[0\ldots \underbrace{1}_{i}\ldots 0]^T$$. So, iteratively created new data is just the first $$k-2$$ columns of $$X_1$$. This is similar for $$X_3,X_4$$, ... etc. So, we're just filtering out the columns of $$X_1$$. We could have done this from the beginning, without the need of any iterations.

• Thank you! Could you elaborate a little bit on the rule for the determining of the eigenvectors? – Danipol Aug 3 at 16:00
• For a given matrix, finding the eigenvectors is equivalent to solving the equation $Ax=\lambda x$, for $\lambda,x$. There are plenty of resources in the internet. For example: lpsa.swarthmore.edu/MtrxVibe/EigMat/MatrixEigen.html – gunes Aug 3 at 18:33