Timeline for What is the difference between Permutation Importance and Drop Column Importance?
Current License: CC BY-SA 4.0
19 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Oct 31 at 15:03 | history | bumped | CommunityBot | This question has answers that may be good or bad; the system has marked it active so that they can be reviewed. | |
| Jun 24 at 22:03 | history | bumped | CommunityBot | This question has answers that may be good or bad; the system has marked it active so that they can be reviewed. | |
| Feb 25 at 8:05 | history | bumped | CommunityBot | This question has answers that may be good or bad; the system has marked it active so that they can be reviewed. | |
| Oct 26, 2023 at 10:03 | history | bumped | CommunityBot | This question has answers that may be good or bad; the system has marked it active so that they can be reviewed. | |
| Jun 25, 2023 at 18:03 | history | bumped | CommunityBot | This question has answers that may be good or bad; the system has marked it active so that they can be reviewed. | |
| Feb 25, 2023 at 12:06 | history | bumped | CommunityBot | This question has answers that may be good or bad; the system has marked it active so that they can be reviewed. | |
| Jan 22, 2023 at 3:37 | answer | added | Hao Zhaojun | timeline score: 0 | |
| Aug 17, 2020 at 8:44 | comment | added | Scortchi♦ | ... its marginal contribution isn't the same as its conditional contribution; arguments for calculating one or the other, or something in between, come down to what you're going to use "importance" for. There's some relevant discussion in Groemping (2009), Am. Stat., 63, 4, "Variable Importance Assessment in Regression: Linear Regression versus Random Forest". | |
| Aug 17, 2020 at 8:43 | comment | added | Scortchi♦ | If you're using 'drop column' importance in a step-down procedure, you keep the model from which the least important predictor was excluded & then recalculate importances - repeating until you've reached your target no. predictors or an unacceptable performance. (Clearly that can take a while, especially in large data-sets, & especially if you cross-validate the whole procedure.) Anyway, the fundamental issue's that if you want to call the contribution of a predictor to a model's overall predictive performance its importance, ... | |
| Aug 13, 2020 at 14:30 | history | edited | Luca Giorgi | CC BY-SA 4.0 |
added 2776 characters in body
|
| Aug 12, 2020 at 1:19 | comment | added | Luca Giorgi | That is partially what I am trying to find out right now with my own work, but I don't think drop column in this context would be good for a step-down procedure either. Perhaps I didn't explain it correctly, but drop column drops one feature at a time, and at the next iteration the previously dropped feature is added back to the feature set and the next one is dropped. At the end you are still left with a ranking of the n features and it's up to you to decide how to use it, e.g. pick the top-n and train a top-n model which might perform better (or just be faster to train). | |
| Aug 12, 2020 at 0:31 | comment | added | Scortchi♦ | Perhaps this ought to be the main thrust of your question. 'Drop column' importance would be the most useful if you were following a step-down procedure to arrive at a smaller predictor set. (Would picking the top $n$ predictors ranked by permutation importance be likely to give you an $n$-predictor model with a performance close to that of the best one? - I don't know.) | |
| Aug 11, 2020 at 23:28 | comment | added | Luca Giorgi | The problem is that in any instance I can think of when you would need feature importance (model explanability, minimal set and all-relevant feature selection), removing an important feature because of collinearity with another (or even duplication) seems wrong to me. In each of those cases it would be better to have both rather than none, so I don't understand how/why drop column would be considered better or more accurate, especially wrt correlated features. | |
| Aug 11, 2020 at 22:42 | comment | added | Scortchi♦ | I think what's confusing is that the authors seem to have a prior notion of what feature importance ought to be - according to which some metrics are praised & others found wanting - but they don't share it with their readers. | |
| Aug 11, 2020 at 22:20 | comment | added | Luca Giorgi | @Scortchi-ReinstateMonica, on the contrary, that section leaves me even more confused. Drop Column is supposed to be the most accurate, but if you dupe a column both will have importance 0 (which to me is wrong), while permutation importance handles the situation a bit more gracefully and shares the importance over the two features. And when looking at "Dealing with collinear features" you see that there's even code to handle collinear/duped features so that they are treated as one and their importance is boosted as a group. | |
| Aug 11, 2020 at 21:59 | comment | added | Scortchi♦ | Is your question not answered in the section entitled "The effect of collinear features on importance"? Two strongly correlated features will be used for roughly equal numbers of splits, & permuting either in the out-of-bag validation samples will have about the same effect on predictive performance, hence roughly equal importances. But drop either one & re-fit, & the other takes its place, resulting in a tiny decrease in performance & hence negligible importance. (Drop both together though, & the decrease in performance might be large.) | |
| Aug 11, 2020 at 17:13 | comment | added | user78229 | A couple of quick observations. First, Breiman developed RFs on his laptop with thousands of observations and two to three thousand predictors (or variables) and a few thousand iterations creating mini-trees or mini-models with each iteration. In his discussion of RFs spoke of it wrt two distinct kinds of permutations. For each iteration: observations are bootstrapped (with replacement) and subsets of randomly sampled predictors. In this way a distribution of how each predictor performed was constructed. Like you, I couldn't find lit wrt Drop Column but it sounds like only 50% of an RF. | |
| Aug 11, 2020 at 16:18 | review | First posts | |||
| Aug 11, 2020 at 18:42 | |||||
| Aug 11, 2020 at 16:11 | history | asked | Luca Giorgi | CC BY-SA 4.0 |