I am working on a classification task. To assess which variable to keep for future modeling, I want to use RF to pick only the most import variable in explaining the binary outcome.
Do the Hyper-parameters of the RF have an influence on variable importance? Should I tune the RF before assessing the importance of the variables?
N.B. The model I want to train after having selected the features will not necessarily be RF.
Ideally you should model the data in the best way you can. You can think it as also capturing the possible interactions / relationship between independent and dependent variable and using that to get variable importance.
That being said, there will be some influence on variable importance when for example mtry is changing, so it helps to be aware of that. The discussion paper in this paper might be useful to you, and below I highlight some of the points discussed:
The more trees are trained, the more stable the predictions should be for the variable importance. In order to assess the stability one could train several random forests with a fixed number of trees and check whether the ranking of the variables by importance are different between the forests.
Genuer et al. (2010) [..] conclude that increasing the mtry value leads to much higher magnitudes of the variable importances.
Grömping (2009) [found..] Higher mtry values lead to lower variable importance of weak regressors.
the random forest standard splitting rule is biased when predictor variables vary in their scale. This has also a substantial impact on the variable importance (Strobl et al., 2007).
I guess it makes sense to check that your variable importance in the context of these.
These are the three papers referenced in this article:
geneur et al Pattern Recognition Letters 2010
