2
$\begingroup$

I've trained and tuned Multilayer Perceptron in a binary classification problem with 18 predictors, and would like to get the permutation feature importance (as model agnostic method,suggested in Fisher et al., 2018 as well as this book: Intrepretable Machine Learning).

As far as I understand the interpretation of the FeatureImp function of the IML R-Package the

Range: should be between 1 (feature is not important) - and positive x.

Interpretation: The higher above 1 the more important is the respective feature.
An Example is given in the IML Book in Section 5.5.3 with:

"Features associated with a model error increase by a factor of 1 (= no >change) were not important for predicting cervical cancer. (..) The feature with the highest importance was Hormonal.Contraceptives..years. associated with an error increase of 4.81 after permutation"

However I got unexpected results. My values for the feature importance have a range between 0 and 1

        feature        importance.05        importance        importance.95        permutation.error
1        x4        0.205839416058394        0.218978102189781        0.256934306569343        0.218978102189781
2        x7        0.0262773722627737        0.0875912408759124        0.106569343065693        0.0875912408759124
3        x1        0        0        0        0
4        x2        0        0        0        0
5        x5        0        0        0        0
6        x6        0        0        0        0

Edit 1: Instead of a function I can now provide a toy example below you find the code for the iris dataset. Additionally I included the plot create by it

########################## Preparation & Libraries ############################
#load libraries
library("dplyr")
library("ggplot2")
library("mlbench")    # for hyperparameter tuning
library("caret")      # for hyperparameter tuning
library("tictoc")     # for a performance measure

df1 <- iris %>% rename(
  Class = Species 
) %>% subset(., Class == "versicolor" | Class == "setosa")
df1$Class <- factor(df1$Class)

########################## Caret Preparation ############################
k.folds = 10
allSummary  <- function(data, lev = NULL, model = NULL){
  a1 <- defaultSummary(data, lev, model)
  b1 <- twoClassSummary(data, lev, model)
  c1 <- prSummary(data, lev, model)
  out <- c(a1, b1, c1)
  out
  # return(out)
}
df1.control <- trainControl( # 10 Fold cross validation, repeated 5 times
  method="repeatedcv", 
  number=k.folds, 
  repeats=5,
  classProbs = T,
  summaryFunction=allSummary
  #savePredictions=T
)

########################## Hyperparametertuning NeuralNet (MLP) ############################
df1.tunegrid <- expand.grid(.size=c(1:(ncol(df1)-1)))
metric <- "Accuracy"

set.seed(1337)
tic("MLP DF1, Hyperparameter Startegy 1: Grid Search")
mlp_df1 <- train(Class~., data=df1, method="mlp", metric=metric, tuneGrid=df1.tunegrid, trControl=df1.control)
toc()
print(mlp_df1)
# plot(mlp_df1)
print(mlp_df1$bestTune)

library("iml")
x <- select(df1,-Class)
y <- select(df1,Class)
predictor <- Predictor$new(mlp_df1, data = x, y = y, type = "prob")
hp_size <- as.numeric(mlp_df1$bestTune) # Tuned Hyperparameter Neruon in hiddenlayer
metric = "ce" # allowed losses "ce", "f1", "logLoss", "mae",

imp <- FeatureImp$new(predictor, 
                      loss = metric,
                      compare = "ratio",
                      n.repetitions = 50
)
plot(imp)

enter image description here

Edit 2 - Solution for the toy example: ok it was pretty simple after all. The MLP has 100% Accuracy, meaning the error baseline is 0. The default importance score computation via ratio (which is described as example in the IML-Book) would result in an error: divide by 0. So the package switches to compare = "difference". And for this a range between 0 and 1 should be correct. This still leaves me guessing about our dataset, since the MLP there doesn't achieve 100% accuracy.

References
Molnar, C., Casalicchio, G., & Bischl, B. (2018). iml: An R package for Interpretable Machine Learning. Journal of Open Source Software, 3(26), 786.
Fisher, A., Rudin, C., & Dominici, F. (2018). Model Class Reliance: Variable importance measures for any machine learning model class, from the ‘Rashomon’ perspective.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.