I would like to understand what is the difference between Permutation Importance (as outlined by Breiman in his original paper on Random Forests) and Drop Column Importance. From a cursory look at both methods, they seem to be doing almost the same thing:
Calculate a baseline score by training a model and obtaining some kind of metric (in this case R2), do something to one of the features and then calculate the score again and record the difference between the baseline and updated scores, thus generating a ranking of how much each feature influences the goodness of the model by whatever metric we are using.
The do something part is where the two methods diverge, where for permutation importance we permute one of the features considering independence with respect to the target variable and other predictor features, while instead for drop column we remove the feature entirely and retrain the model before computing the score again.
To me these two operations should give almost the same result, with the disadvantage in drop column of having to train the model P times (no. of features). Of course this is not the case, as per this blog post (emphasis mine):
Permutation importance does not require the retraining of the underlying model [...], this is a big performance win. The risk is a potential bias towards correlated predictive variables.
If we ignore the computation cost of retraining the model, we can get the most accurate feature importance using a brute force drop-column importance mechanism.
However I don't understand why the act of permutating (permuting?) one of the features, with the explicit consideration of independence between both the target variable and the other predictors, would give a bias towards correlated features.
I am aware of Strobl's work on a conditional permutation scheme which tackles specifically this issue, however it does so by modifying the null hypothesis under which the permutation is performed and considering indepence only between the feature and the target, which to me, as you can guess, is counterintuitive.
What I also don't understand is if, bias towards correlated feature notwithstanding, the two algorithms are equivalent, so to speak.
P.S.: On a side note, I would also like to know if there's a source in literature for the drop column approach, as I am including it in a text I'm currently writing and don't know how to reference it since I couldn't find an original source.
EDIT:
I now better understand why Strobl et al.'s proposed conditional permutation scheme is less biased towards correlated variables, this passage from their paper (Conditional variable importance for random forests published in BCM Bioinformatics) was quite clear:
What is crucial when we want to understand why correlated predictor variables are preferred by the original random forest permutation importance is that a positive value of the importance corresponds to a deviation from this null hypothesis – that can be caused by a violation of either part: the independence of $X_j$ and $Y$, or the independence of $X_j$ and $Z$ [other predictor variables, ed.]. However, from these two aspects only one is of interest when we want to assess the impact of $X_j$ to help predict $Y$, namely the question if $X_j$ and $Y$ are independent. This aim, to measure only the impact of $X_j$ on $Y$, would be better reflected if we could create a measure of deviation from the null hypothesis that $X_j$ and $Y$ are independent under a given correlation structure between $X_j$ and the other predictor variables, that is determined by our data set. To meet this aim we suggest a conditional permutation scheme, where $X_j$ is permuted only within groups of observations with $Z$ = $z$, to preserve the correlation structure between $X_j$ and the other predictor variables [...]
My question still stands as to what differs between the original permutation scheme and the drop column technique.
Moreover, as highlighted in the comments, in the extreme example of a duplicate column in the data the drop column algorithm will assign a score of 0 to both (due to how importance is computed in this case), and while this is certainly not biased towards correlated features it removes the importance of a potentially very influential column from the results. What's more, since the importance metric is computed from the underlying random forest (which as we have established has inherent biases), when we have two strongly correlated columns and drop one of the two the modell will still be able to obtain some of that information from the correlated feature still present in the data set, and thus there (potentially) won't be a large difference between the baseline score and the score computed when dropping either feature, meaning that both correlated features will have a lower overall importance score at the end.
This to me is in stark contrast to the "most accurate feature importace" that we are supposed to get from this very computationally expensive algorithm, and thus I'm even more confused as to what are the benefits of drop column over the original permutation scheme, not to mention the conditional variation by Strobl et al.
