Random Forest - Variable Importance over time

Question

I've been looking at measures of variable importance for a random forest model - and was wondering if there are ways in which you can track how the variable importance shifts as the model is applied to datasets in the future.

The purpose of this question is to gain information as to whether a variable that was very important at model development is no longer that important when making predictions on future datasets either due to a change in population or a change in the variable itself (i.e. if this variable was somehow distorted and replaced with a column of missings it would no longer be important!).

Some example code (sourced from : How to interpret Mean Decrease in Accuracy and Mean Decrease GINI in Random Forest models) may help illustrate the problem:

require(randomForest)
data(iris)
set.seed(1)
dat <- iris
dat$Species <- factor(ifelse(dat$Species=='virginica','virginica','other'))
model.rf <- randomForest(Species~., dat, ntree=25,
importance=TRUE, nodesize=5)
model.rf
varImpPlot(model.rf)

It seems that the variable importance plot can only be created on the dataset in which the random forest model was trained on as the variable importance plot function can only be applied to a random forest object (which stays the same no matter what dataset it is trying to score in the future).

Is there a built-in way (in Python or R) to compute variable importance over time, or is it not possible for some reason? My understanding is that if a new dataset possessed an outcome flag it would be possible to compute the mean decrease in gini.

Edit: Would also like to slightly expand this question to discuss potential ways on measuring variable stability over time. For example, in a standard logistic scorecard one would compute a characteristic stability index using the pre-defined bins. However, as many random forests have continuous inputs the choice of bins isn't natural and potentially there should be an alternative method?

Answer 1

Permutation importances would be suited for your goal, you can compute them as follows (example R code apended below):

1. Fit a RF on a training dataset.
2. Compute MSE on test observations.
3. For each predictor variable, randomly permute the values of that variable in the test set and compute the MSE again. The difference between this MSE and the MSE computed under 2. is the variables' contribution to the MSE reduction of the RF.

Note that this approach will not yield identical importance estimates for the training data as the default ones, also because there are many different ways variable importances are standardized, see also documentation of e.g., randomForest.

R code example:

## Generate toy data:
set.seed(42)
x1 <- rnorm(1000)
x2 <- rnorm(1000)
y <- 1*x1 + 2*x2 + rnorm(1000)
train <- data.frame(x1 = x1[1:500], x2 = x2[1:500], y = y[1:500])
test <- data.frame(x1 = x1[501:1000], x2 = x2[501:1000], y = y[501:1000])

## Fit random forest:
library("randomForest")
set.seed(42)
RF <- randomForest(y ~ x1 + x2, data = train, importance = TRUE)
importance(RF)

## Compute MSEs:
MSE_train <- mean((train$y - predict(RF, newdata = train))^2)
MSE_test <- mean((test$y - predict(RF, newdata = test))^2)
pred_names <- c("x1", "x2")
perm_MSEs_train <- perm_MSEs_test <- c(NA, NA)
names(perm_MSEs_train) <- names(perm_MSEs_test) <- pred_names

## Compute permutation importances:
for (i in pred_names) {
  test_temp <- test
  test_temp[ , i] <- sample(test[ , i])
  perm_MSEs_train[i] <- mean((test$y - predict(RF, newdata=test_temp))^2)
  train_temp <- train
  train_temp[ , i] <- sample(train[ , i])
  perm_MSEs_test[i] <- mean((train$y - predict(RF, newdata=train_temp))^2)
}
var(train$y)
MSE_train ## Note that overfitting occurred quite a bit: MSE should be 1
perm_MSEs_train - MSE_train
var(test$y)
MSE_test
perm_MSEs_test - MSE_test

Answer 2

My current solution is simply to retrain the model on newer data (but keeping the tuning parameters the same as development). This way the built in "importance" function can calculate updated variable importances.

This method is not optimal as it does not necessarily explain the causes behind a shift in predictions from the actual model that is being used (i.e. the one made at development).

Hence, would still appreciate any answers that could help with the original question.

Answer 3

I believe that the reason why we don't have the option to calculate variable importance on newer data is more a computational/complexity issue rather than actual practicality?

This is because when training the Random Forest model we are already looking at decrease in Gini to optimise splits so essentially we are getting variable importance information for free when training the model. Whereas to calculate mean decrease in gini on newer data would require us to go back into the trees and almost repeat the training procedure.

Also as mean decrease in accuracy (which is in the same importance function) uses the "Out Of Bag" data which is not possible on new data that isn't used for training - I guess adding the option to calculate importance on a dataset other than the development one wasn't considered.