Identify original features corresponding to high singular/principal component values [duplicate]

Question

In MATLAB, while SVD gives a diagonal matrix S of decreasing singular values, PCA gives a column vector LATENT of decreasing principal component values.

1. How can S be used to obtain a subset of original features that best describe the data?
2. How can LATENT be used to obtain a subset of original features that best describe the data?

NOTE: QR decomposition in MATLAB gives a permutation vector E which enables the above.

Answer 1

To select the important variables (features) in your dataset, you will need more than the output you have described. These output are useful for selection procedures for finding the number of components, but not the necessary variables. S corresponds to singular values (the square root of the eigenvalues), and LATENT to eigenvalues. Thus, they give a sense of the amount of variation captured by the dimensions of the transformed variables.

A more useful output are the eigenvectors. These are V in the SVD output. These are the COEFF output from PCA. The elements of the eigenvectors are the coefficients in the linear relationships defined by PCA. Thus a variable that has very low values for all of the retained eigenvectors is potentially excludeable. The simplest way to exclude unnecessary variables is to set a threshold: if the variable has coefficients that are all below a threshold, it is dropped. This is called simple thresholdling. However, Cadima and Jolliffe (1995) show that the results can be misleading

• Cadima, J. and Jolliffe, I. T. (1995). Loadings and Correlations in the Interpretation of Principal Components. Journal of Applied Statistics, 22(2), 203–214.

A better approach would be to try a sparse PCA method. Two are: SCoTLASS, detailed in the paper below:

• Jolliffe, I. T., Trendafilov, N. T., and Uddin, M. (2003). A Modified Principal Component Technique Based on the LASSO. Journal of Computational and Graphical Statistics, 12(3), 531–547.

and SPCA, introduced in the following paper:

• Zou, H., Hastie, T., and Tibshirani, R. (2006). Sparse Principal Component Analysis. Journal of Computational and Graphical Statistics, 15(2), 265–286.

SPCA is implemented in a MATLAB toolbox. I would recommend trying spca() in the Matlab toolbox with the "soft thresholding" option to see if it helps (see example in the spca code).

If you do not use soft thresholding (an automatic, data driven sparsity parameter selection procedure) you will have to select the sparsity parameters yourself. Consult the SCoTLASS paper or the SPCA paper for guidelines on parameter selection. You will want to select a sparsity parameter that excludes unnecessary variables, but is not so strict that important ones are also dropped.

After you have performed a sparse PCA procedure, the unnecessary variables should hopefully have very low coefficient values (typically low enough that you can safely apply an epsilon small threshold without encountering the problems highlighted in Cadima '95) and you will be able to clearly identify variables that are important in your data.