I have a rather high-dimensional data set (p > 1000) with several variables ranking significantly higher than the rest in terms of variable importance (measured by Gini impurity).
However, these variables are highly correlated and thus it is unclear whether each variable actually holds unique information or simply ranks high due to correlation to the causal variable.
My general approach for testing the significance of the importance of variables is a bootstrap permutation test, i.e. I first bootstrap from the training data and build a random forest model and then permute the columns in the out-of-bootstrap samples and check if I observe a significant decline in accuracy.
In these cases my assumption under H0 is that these variables are completely uninformative and thus it shouldn't matter whether I permute the column of the training or test set.
However, in the above case, the assumption does not hold. The variables are predictive and and due to sub-sampling of variables at each split, they will get selected during model training. If I now permute the corresponding columns of the test set, it is clear that the performance will drop, although this does not indicate whether these variables hold unique information at all.
Therefore, I'm thinking about changing my approach in the sense that I'm permuting the variables in the training set and not in the test set.
My question is: Is this approach valid? Is permutation of variables in the training set a better approach in general? If not, in which cases would it be not appropriate?