PCA: Iteratively reducing dimensions ist there a difference between Iteratively removing one dimension with the least variance by computing pca and repeating this process until the data has k dimensions less...
Compared to calculating pca and removing the k dimensions with the least variance at once.
Thank you!
 A: I believe there are shorter explanations, but here is my bit on it: PCA component axes are eigenvectors of the data covariance, and if $u$ is a normalized eigenvector, $-u$ is also one. Since you can choose one of them arbitrarily, iterative PCA does not guarantee the same output. However, if we put a rule on the eigenvectors such that they're uniquely determined, e.g. the first non-zero element will be positive, the two procedures will produce same results.
Without loss of generality, we can assume that our data is centered, i.e. has zero mean; let's call it as $X_{n\times k}$, where $n$ is the number of samples and $k$ is the number of dimensions. Let the eigen decomposition of $X^TX$ be $US_kU^T$, where the diagonal elements of $S_k$ are in non-increasing order. The first $k-1$ columns of $U$ will define the first $k-1$ component axes. When we omit the last column, i.e. the column with the least explained variance, the new data is $X_1=X\ [u_1\ldots u_{k-1}]$, which also has zero mean.
Now, we apply PCA again on this one: $$\begin{align}X_1^TX_1&=\begin{bmatrix} u_1^T \\ \vdots \\u_{k-1}^T \end{bmatrix}X^TX\begin{bmatrix}u_1 \ldots u_{k-1}\end{bmatrix}=\begin{bmatrix} u_1^T \\ \vdots \\u_{k-1}^T \end{bmatrix}US_kU^T\begin{bmatrix}u_1 \ldots u_{k-1}\end{bmatrix}=\underbrace{S_{k-1}}_{U_1S_{k-1}U_1^T,\ \ U_1=I}\end{align}$$
$S_{k-1}$ denotes the upper-left $k-1\times k-1$ part of $S_k$, containing the largest $k-1$ eigenvalues. So, the new data scatter matrix is already in diagonal form, which means the eigenvectors are canonical vectors, i.e. $U_1=I_{k-1}$. When we do one more iteration, we calculate the following:
$$X_2=X_1[e_1\ldots e_{k-2}]=[x_{1,1}\ldots x_{1,k-1}]$$
where $x_{1,j}$ denotes the $j$-th column of $X_1$, and $e_i$ is the canonical vector, i.e. $e_i=[0\ldots \underbrace{1}_{i}\ldots 0]^T$. So, iteratively created new data is just the first $k-2$ columns of $X_1$. This is similar for $X_3,X_4$, ... etc. So, we're just filtering out the columns of $X_1$. We could have done this from the beginning, without the need of any iterations.
