Choosing the dimension in PCA [duplicate]

Question

Suppose that a dataset $(x_1, \cdots x_N) =X \in \mathbb R^{d \times N}$ gives rise to the PCA (principal components analysis) result $\tilde X^\top \tilde X = U^\top DU$ where $\tilde X = (x_1 - \bar x, \cdots x_N - \bar x)$ is the centered matrix with $\bar x = \frac1N (x_1 + \cdots + x_N)$, $U$ is a symmetric matrix, and $D = \text{diag}(\lambda_1, \cdots \lambda_N)$ is a diagonal matrix of the principal values. Here, we may assume that $\lambda_1 \ge \lambda_2 \ge \cdots$.

Question. How does one choose a number $k$ representing the intrinsic dimension of $X$ so that we only use the top $k$ principal components (the column vectors from $U$)?

It seems that there are a few ways to do this, for example by taking the maximum $k$ such that $\lambda_k \ge \epsilon$, where $\epsilon \ge0$ is an error threshold. Analogously, one can bound the tail sum instead: $\lambda_{k+1} + \cdots + \lambda_N < \epsilon$, or the $L^2$ analogue: $\sqrt{\lambda_{k+1}^2 + \cdots + \lambda_N^2} < \epsilon$. There are more involved methods too, such as Minka's NeurIPS paper. However, I am curious of what is the most canonical method, either in practice or in theory.

Answer 1

It's customary to set $k=\#\{\lambda_j>1\}$, that is the number of eigenvalues greater than one. Other approches can be taken, which I use a lot, but I also use the greater-than-one criteria a lot.

Mathematically, the notation can also be written in the form:

$k=\sum_{j=1}^N I(\lambda_j>1)$

where $I=1$ if $\lambda_j>1$, and zero otherwise.

Answer 2

An approach for choosing the number of dimensions to keep is to consider the amount of variance explained by each principal component.

We know that the variance of the data is equal to the diagonal sum of matrix $D$, i.e. the sum of the eigenvalues $Var(X)=\sum_{I} \lambda_{I}$. However, each eigenvalue corresponds to a principal component.

So, you can put in decreasing order the eigenvalues and start calculating their cumulative sum. Continue calculating the cumulative sum until there is no significant change into the sum.

For example, let $\lambda_{1}=15,\lambda_{2}=2,\lambda_{3}=0.2,\lambda_{4}=10$. Then $\lambda_{1}>\lambda_{4}>\lambda_{2}>\lambda_{3}$. You start calculating the cumulative sum $\lambda_{1}+\lambda_{4}=25$, then you calculate $\lambda_{1}+\lambda_{4}+\lambda_{2}=27$. So, the variance explained by the pricipnal component that corresponds to $\lambda_{2}$ is small. So, we can keep only two principal components the ones that corresponds to $\lambda_{1}$ and $\lambda_{4}$.