# Feature selection and latent variables

I would like to know if it is useful (or maybe dangerous) to reduce the number of attributes (by selecting the most informative ones among thousands) before seeking for latent variables or not (in an exploratory perspective).

A subsidiary question: in the same case, would it be beneficial to select the most important features for each categorie of features (these can be compressed using an entity-attribute-value model which is not really suitable for data mining) before detecting the latent variables?

Briefly, the answer is maybe (It is hard for me to tell what you are going to do with your data after variable selection has been performed). It is known that for least squares or PLS regression, feature selection should be performed prior to construction of the final model. The reason for this is that the mean squared error of prediction for these models has a term which depends on $(p/n)^2$ where $p$ is the number of 'attributes', and $n$ is the number of observations (or samples). The number of latent factors used in PLS is evidently immaterial in this result. In my view, this somewhat undermines the raison d'etre of PLS, but I still find it useful.
• I've already +1, but forgot to point to the following paper by Chun and Keleş: Sparse Partial Least Squares Regression for Simultaneous Dimension Reduction and Variable Selection (JRSS B, 2010, 72(1):3–25). Basically, it shows that in the $n\ll p$ case, consistency of the PLS estimator no longer holds, and the authors proposed an L1 penalty (based on the LARS algorithm) to the original Wold's algorithm. A working R package can be found on CRAN: spls. – chl Jul 11 '11 at 21:52