Timeline for Are important features or noise model agnostic?
Current License: CC BY-SA 4.0
10 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Mar 19, 2022 at 0:18 | answer | added | Yves-Laurent Kom Samo | timeline score: 0 | |
| Mar 16, 2022 at 3:24 | comment | added | Dave | What are “noisy features”, features that you suspect to be unrelated to the outcome? | |
| Mar 11, 2022 at 19:33 | comment | added | Sycorax♦ | Perhaps you could edit your question to clarify what you know, you’d like to know about and where you are stuck. Overall, I'm not really a big believer in feature selection, because it attempts to replace knowing about your data and the problem you're solving with an algorithmic procedure, which seems like a fool's errand to me. | |
| Mar 11, 2022 at 18:20 | comment | added | Shubham Agrawal | Thanks for responding, I see your point. So, if I want to try to different algorithms on the dataset, I should select informative features by CV for each algorithm separately using that algorithm only? Also, do you think it's a valid approach e.g. if I select the relevant features using default SVM with linear kernel for dimensionality reduction, and then tune the hyperparameters for the SVM, once the features are selected? Can we at least say for sure that features selected from a different setting of the same algorithm are relevant for all other settings? | |
| Mar 11, 2022 at 18:02 | comment | added | Sycorax♦ | Nothing you've quoted in any way disproves the claim made in the linked thread. The core claim -- that you can go from a larger number of features to a smaller number -- is vacuously true. There's no guarantee the selected features are relevant. But using a linear model to select features when the data-generating process is nonlinear can be disastrous, as demonstrated in the link I shared. | |
| Mar 11, 2022 at 12:09 | comment | added | Shubham Agrawal | @Sycorax Also this, from the same page: "we make use of a LinearSVC coupled with SelectFromModel to evaluate feature importances and select the most relevant features. Then, a RandomForestClassifier is trained on the transformed output, i.e. using only relevant features. You can perform similar operations with the other feature selection methods and also classifiers that provide a way to evaluate feature importances of course. " | |
| Mar 11, 2022 at 11:24 | comment | added | Shubham Agrawal | @Sycorax This is from scikit-learn's documentation: "Linear models penalized with the L1 norm have sparse solutions: many of their estimated coefficients are zero. When the goal is to reduce the dimensionality of the data to use with another classifier, they can be used along with SelectFromModel to select the non-zero coefficients. In particular, sparse estimators useful for this purpose are the Lasso for regression, and of LogisticRegression and LinearSVC for classification". Is this correct? | |
| Mar 6, 2022 at 20:35 | comment | added | Christian Hennig | At least once features are not independent, features cannot cleanly be partitioned into "important features" and "noise". For example, if feature X1 and X2 share the same information about the outcome, any of these is unimportant given the other, but one of them is needed (unless the same information is in another feature). Different approaches may handle such cases differently (and in reality information shared by various features is rather the rule than the exception). | |
| Mar 6, 2022 at 19:17 | comment | added | Sycorax♦ | Whether or not a feature is informative depends on the model. Here's an example, comparing random forest and logistic regression on a toy problem: stats.stackexchange.com/questions/164048/… | |
| Mar 6, 2022 at 19:13 | history | asked | Shubham Agrawal | CC BY-SA 4.0 |