I've been looking at measures of variable importance for a random forest model - and was wondering if there are ways in which you can track how the variable importance shifts as the model is applied to datasets in the future.
The purpose of this question is to gain information as to whether a variable that was very important at model development is no longer that important when making predictions on future datasets either due to a change in population or a change in the variable itself (i.e. if this variable was somehow distorted and replaced with a column of missings it would no longer be important!).
Some example code (sourced from : How to interpret Mean Decrease in Accuracy and Mean Decrease GINI in Random Forest models) may help illustrate the problem:
require(randomForest)
data(iris)
set.seed(1)
dat <- iris
dat$Species <- factor(ifelse(dat$Species=='virginica','virginica','other'))
model.rf <- randomForest(Species~., dat, ntree=25,
importance=TRUE, nodesize=5)
model.rf
varImpPlot(model.rf)
It seems that the variable importance plot can only be created on the dataset in which the random forest model was trained on as the variable importance plot function can only be applied to a random forest object (which stays the same no matter what dataset it is trying to score in the future).
Is there a built-in way (in Python or R) to compute variable importance over time, or is it not possible for some reason? My understanding is that if a new dataset possessed an outcome flag it would be possible to compute the mean decrease in gini.
Edit: Would also like to slightly expand this question to discuss potential ways on measuring variable stability over time. For example, in a standard logistic scorecard one would compute a characteristic stability index using the pre-defined bins. However, as many random forests have continuous inputs the choice of bins isn't natural and potentially there should be an alternative method?