I've been looking at measures of variable importance for a random forest model - and was wondering if there are ways in which you can track how the variable importance shifts as the model is applied to datasets in the future.

The purpose of this question is to gain information as to whether a variable that was very important at model development is no longer that important when making predictions on future datasets either due to a change in population or a change in the variable itself (i.e. if this variable was somehow distorted and replaced with a column of missings it would no longer be important!).

Some example code (sourced from : How to interpret Mean Decrease in Accuracy and Mean Decrease GINI in Random Forest models) may help illustrate the problem:

dat <- iris
dat$Species <- factor(ifelse(dat$Species=='virginica','virginica','other'))
model.rf <- randomForest(Species~., dat, ntree=25,
importance=TRUE, nodesize=5)

It seems that the variable importance plot can only be created on the dataset in which the random forest model was trained on as the variable importance plot function can only be applied to a random forest object (which stays the same no matter what dataset it is trying to score in the future).

Is there a built-in way (in Python or R) to compute variable importance over time, or is it not possible for some reason? My understanding is that if a new dataset possessed an outcome flag it would be possible to compute the mean decrease in gini.

Edit: Would also like to slightly expand this question to discuss potential ways on measuring variable stability over time. For example, in a standard logistic scorecard one would compute a characteristic stability index using the pre-defined bins. However, as many random forests have continuous inputs the choice of bins isn't natural and potentially there should be an alternative method?


My current solution is simply to retrain the model on newer data (but keeping the tuning parameters the same as development). This way the built in "importance" function can calculate updated variable importances.

This method is not optimal as it does not necessarily explain the causes behind a shift in predictions from the actual model that is being used (i.e. the one made at development).

Hence, would still appreciate any answers that could help with the original question.

| cite | improve this answer | |

I believe that the reason why we don't have the option to calculate variable importance on newer data is more a computational/complexity issue rather than actual practicality?

This is because when training the Random Forest model we are already looking at decrease in Gini to optimise splits so essentially we are getting variable importance information for free when training the model. Whereas to calculate mean decrease in gini on newer data would require us to go back into the trees and almost repeat the training procedure.

Also as mean decrease in accuracy (which is in the same importance function) uses the "Out Of Bag" data which is not possible on new data that isn't used for training - I guess adding the option to calculate importance on a dataset other than the development one wasn't considered.

| cite | improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.