1
$\begingroup$

ist there a difference between Iteratively removing one dimension with the least variance by computing pca and repeating this process until the data has k dimensions less...

Compared to calculating pca and removing the k dimensions with the least variance at once.

Thank you!

$\endgroup$
1
$\begingroup$

I believe there are shorter explanations, but here is my bit on it: PCA component axes are eigenvectors of the data covariance, and if $u$ is a normalized eigenvector, $-u$ is also one. Since you can choose one of them arbitrarily, iterative PCA does not guarantee the same output. However, if we put a rule on the eigenvectors such that they're uniquely determined, e.g. the first non-zero element will be positive, the two procedures will produce same results.

Without loss of generality, we can assume that our data is centered, i.e. has zero mean; let's call it as $X_{n\times k}$, where $n$ is the number of samples and $k$ is the number of dimensions. Let the eigen decomposition of $X^TX$ be $US_kU^T$, where the diagonal elements of $S_k$ are in non-increasing order. The first $k-1$ columns of $U$ will define the first $k-1$ component axes. When we omit the last column, i.e. the column with the least explained variance, the new data is $X_1=X\ [u_1\ldots u_{k-1}]$, which also has zero mean.

Now, we apply PCA again on this one: $$\begin{align}X_1^TX_1&=\begin{bmatrix} u_1^T \\ \vdots \\u_{k-1}^T \end{bmatrix}X^TX\begin{bmatrix}u_1 \ldots u_{k-1}\end{bmatrix}=\begin{bmatrix} u_1^T \\ \vdots \\u_{k-1}^T \end{bmatrix}US_kU^T\begin{bmatrix}u_1 \ldots u_{k-1}\end{bmatrix}=\underbrace{S_{k-1}}_{U_1S_{k-1}U_1^T,\ \ U_1=I}\end{align}$$

$S_{k-1}$ denotes the upper-left $k-1\times k-1$ part of $S_k$, containing the largest $k-1$ eigenvalues. So, the new data scatter matrix is already in diagonal form, which means the eigenvectors are canonical vectors, i.e. $U_1=I_{k-1}$. When we do one more iteration, we calculate the following: $$X_2=X_1[e_1\ldots e_{k-2}]=[x_{1,1}\ldots x_{1,k-1}]$$ where $x_{1,j}$ denotes the $j$-th column of $X_1$, and $e_i$ is the canonical vector, i.e. $e_i=[0\ldots \underbrace{1}_{i}\ldots 0]^T$. So, iteratively created new data is just the first $k-2$ columns of $X_1$. This is similar for $X_3,X_4$, ... etc. So, we're just filtering out the columns of $X_1$. We could have done this from the beginning, without the need of any iterations.

$\endgroup$
  • $\begingroup$ Thank you! Could you elaborate a little bit on the rule for the determining of the eigenvectors? $\endgroup$ – Danipol Aug 3 at 16:00
  • $\begingroup$ For a given matrix, finding the eigenvectors is equivalent to solving the equation $Ax=\lambda x$, for $\lambda,x$. There are plenty of resources in the internet. For example: lpsa.swarthmore.edu/MtrxVibe/EigMat/MatrixEigen.html $\endgroup$ – gunes Aug 3 at 18:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.