4
$\begingroup$

I would like to know if it is useful (or maybe dangerous) to reduce the number of attributes (by selecting the most informative ones among thousands) before seeking for latent variables or not (in an exploratory perspective).

A subsidiary question: in the same case, would it be beneficial to select the most important features for each categorie of features (these can be compressed using an entity-attribute-value model which is not really suitable for data mining) before detecting the latent variables?

Improve this question
$\endgroup$

2 Answers 2

Reset to default
1
$\begingroup$

Briefly, the answer is maybe (It is hard for me to tell what you are going to do with your data after variable selection has been performed). It is known that for least squares or PLS regression, feature selection should be performed prior to construction of the final model. The reason for this is that the mean squared error of prediction for these models has a term which depends on $(p/n)^2$ where $p$ is the number of 'attributes', and $n$ is the number of observations (or samples). The number of latent factors used in PLS is evidently immaterial in this result. In my view, this somewhat undermines the raison d'etre of PLS, but I still find it useful.

Improve this answer
$\endgroup$
2
  • $\begingroup$ I've already +1, but forgot to point to the following paper by Chun and Keleş: Sparse Partial Least Squares Regression for Simultaneous Dimension Reduction and Variable Selection (JRSS B, 2010, 72(1):3–25). Basically, it shows that in the $n\ll p$ case, consistency of the PLS estimator no longer holds, and the authors proposed an L1 penalty (based on the LARS algorithm) to the original Wold's algorithm. A working R package can be found on CRAN: spls. $\endgroup$
    – chl
    Jul 11, 2011 at 21:52
  • $\begingroup$ @chl: nice catch. I'm putting that paper on my queue ;) $\endgroup$
    – shabbychef
    Jul 13, 2011 at 17:32
0
$\begingroup$

How dangerous this activity is depends upon the properties of your features and the tests that you use to reduce the feature number. It may be the case that you will not find the "one-test-fits-all" method for reducing features, in which case its important to understand and state the assumptions you're making when performing the test.

It is hard from your statement to judge what the purpose of the latent variables is in your analysis. However, depending upon your data, feature selection is useful because too much data may serve to dampen your true signal and add noise to classification.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.