Timeline for Must we do feature selection in cross validation?
Current License: CC BY-SA 4.0
7 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Sep 29, 2021 at 7:35 | comment | added | nan | @Tim Thanks for the detailed answer, no mine is more of classification, am just worried if logistic regression will also need to obey the multi-collinearity issue | |
| Sep 29, 2021 at 7:07 | comment | added | Tim | @nan in general you do not need to remove any features. In machine learning, the models can deal with "selecting" the features they use by themselves. You can use regularization to facilitate it even more. You remove the features if you have really good reasons for that (e.g. the former didn't work). On another hand, if using linear regression for inference, you need to bother about perfect multicollinearity, and the features you select should be based on your research question, not exploratory data analysis and cherry-picking. | |
| Sep 29, 2021 at 7:00 | comment | added | nan | Dear both, thanks for the insights. There are a few features with 0.9 and above in correlation. So I am stuck in between whether I should justify why I need to remove those correlated features, or if there is any other way to help me choose the features I want. Is there any good methods to help me select the features? I have seen people using a tree to fit the model and get the feature_importance_ from them, and selecting the top N features, and retrain it again, is this recommended? | |
| Sep 29, 2021 at 6:55 | comment | added | Dikran Marsupial | +1 Performing the feature selection manually is probably worse as the "researcher degrees of freedom" are difficult to account for in the performance evaluation (and in describing exactly what you did, so it is difficult to make the work reproducible). We ought to avoid "CyborgML" (AutoML where the researcher becomes part of the mechanism) where possible ;o) I always dread to see "we determined the hyper-parameters via a preliminary exploration" (or similar) when I am reviewing a paper, as is means the results are likely to be questionable (due to CyborgML overfitting). | |
| Sep 29, 2021 at 6:55 | comment | added | Tim | @nan you mean perfect multicollinearity or just correlation between features? Also, it's something you need to bother about it when using linear regression, but not in case of most of the machine learning models. | |
| Sep 29, 2021 at 6:50 | comment | added | nan | Thanks for this, I have updated my question to ask whether multi-collinearity features should be handled before cross-validation? | |
| Sep 29, 2021 at 6:47 | history | answered | Tim | CC BY-SA 4.0 |