I read from literature that the following two methods can be used for feature selection prior to model development: 1. Correlation factor between target and feature variables (select those features that have correlation > threshold) 2. Lasso

Which of the above two methods is preferred?

In one of the exercises I did, Lasso retained some features which have a lower correlation than the features it dropped. In other words, the above two methods didn't result in the same set of features selected. How do we explain this?

  • $\begingroup$ Lasso acts on the conditional (i.e., partial) correlation between features and the target, whereas the correlation method acts on the marginal correlation between the features and the target. The partial correlation is more relevant for prediction since you will be using all the variables you end up including in future predictions, so I would expect the lasso method to be a better choice. $\endgroup$ – Noah Nov 25 '19 at 7:19
  • $\begingroup$ Selecting features based on correlations is dubious, (the whole correlation does not equal causation) because 1) the correlation may not be linear or monotonous (Pearson / Spearman), 2) there may be intercorrelation between the variables, which you will not identify with a correlation coefficient. $\endgroup$ – user2974951 Nov 26 '19 at 12:23

I think by saying correlation you are referring to SIS, developed by Jianqing Fan and Jinchi Lv. Actually, the logic behind the two methods is different. LASSO does the selection by using a penalized loss function and sparsity of the variables is required. Normally, for ultra-high dimensional data, we perform SIS first and reduce the dimension to a relatively small amount, and then perform LASSO to further reduce the number of variables that enter the final model.

  • $\begingroup$ Thanks... by Correlation I meant Pearson correlation factor between the target (output) and input variables. $\endgroup$ – Lakshmi Srinivasan Nov 26 '19 at 15:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.