Component selection PCA

Question

I have just started learning about machine learning algorithms. My senior at work tasked me to learn about dimensionality reduction using principle component analysis. In that he said that the number of components used to redefine the larger dimension raw data can be selected by calculating the reduction in eigen values of the components(slope between the 1st PCA and 2nd PCA,2nd PCA and 3rd PCA and so on) and using a particular threshold. He wants me to find the optimal threshold. I searched online but there was no references about this particular method of component selection. I found other methods like scree plot, cumulative variance, eigen one threshold etc,. Does anyone know anything about this ? Please help.

Answer 1

I reckon that you haven't done your homework carefully.

The criterion for component extraction is well written in many papers. A popular rule of thumb is selecting eigenvalues larger than 1. Besides, there are also criteria based on boostrapping and cross-validation. You can read in more detail in (Wold, Esbensen, & Geladi, 1987) and cited papers.

References

Wold, S., Esbensen, K., & Geladi, P. (1987). Principal Component Analysis. Chemometrics and Intelligent Laboratory Systems, 2(1-3), 37–52. doi:10. 1016/0169-7439(87)80084-9

Answer 2

In fact you are right being confused - there is no such thing as a generally optimal threshold. All available criteria are basically arbitrary rules of thumb given to people who need a formal rule. Obviously if you have nothing else to go by, you can use one of them, see the answer by little_monster.

In fact there are many situations in which how many PCs to choose is very ambiguous. What is even meant by "optimality" depends on what the PCs are used for. For example, if you use PCs for principal component regression, you can use things such as cross-validation to choose the number of PCs that is best at predicting the response variable. But in general it's always a trade-off between low dimensionality and explained variance, and what is more important to you or how many more PCs you accept for explaining what higher percentage of the variance cannot in general be "optimally" decided.