I wished I could post a reproducible example but the following is observed in a specific large data set that I cannot share. A set of variables shows consistently high variable (permutation based) importance in random forests. However, leaving out those variables leads to unchanged OOB and test MSE. Note that replacing these variables with their residuals from a regression (or even a RF) on all the other variables does not change the variable importance ranking much. I cannot think of a sound explanation of this apparent contradiction.
I cannot think of a sound explanation of this apparent contradiction.
Those variables might achieve supreme perfection at separating the classes.
Nevertheless, it is possible that the remaining variables after you drop those supreme ones are still positively excellent. Therefore you wouldn't see a difference because your data-set is just very easy to classify.
Random forest is robust. Think of it as an ML analog to the median. If you remove one point in an ensemble, it doesn't impact the median much.
How do you measure your importance? If you have many thousand rows, and a p of 1%, then you might have a problem. I would look at the plot of importances, and cluster within it.
It may be that the "important" variable can be, in a moneyball sense, "reconstructed in the aggregate" from other columns that are noisier. Those columns will show up as less important by themselves, but when the most important is missing, they will be brought into play.
If you can give a little detail on how you determined importance, that might help.