# Permutation Variable Importance in MLP - Unexpected Results with IML R-Package

I've trained and tuned Multilayer Perceptron in a binary classification problem with 18 predictors, and would like to get the permutation feature importance (as model agnostic method,suggested in Fisher et al., 2018 as well as this book: Intrepretable Machine Learning).

As far as I understand the interpretation of the FeatureImp function of the IML R-Package the

Range: should be between 1 (feature is not important) - and positive x.

Interpretation: The higher above 1 the more important is the respective feature.
An Example is given in the IML Book in Section 5.5.3 with:

"Features associated with a model error increase by a factor of 1 (= no >change) were not important for predicting cervical cancer. (..) The feature with the highest importance was Hormonal.Contraceptives..years. associated with an error increase of 4.81 after permutation"

However I got unexpected results. My values for the feature importance have a range between 0 and 1

        feature        importance.05        importance        importance.95        permutation.error
1        x4        0.205839416058394        0.218978102189781        0.256934306569343        0.218978102189781
2        x7        0.0262773722627737        0.0875912408759124        0.106569343065693        0.0875912408759124
3        x1        0        0        0        0
4        x2        0        0        0        0
5        x5        0        0        0        0
6        x6        0        0        0        0


Edit 1: Instead of a function I can now provide a toy example below you find the code for the iris dataset. Additionally I included the plot create by it

########################## Preparation & Libraries ############################
library("dplyr")
library("ggplot2")
library("mlbench")    # for hyperparameter tuning
library("caret")      # for hyperparameter tuning
library("tictoc")     # for a performance measure

df1 <- iris %>% rename(
Class = Species
) %>% subset(., Class == "versicolor" | Class == "setosa")
df1$$Class <- factor(df1$$Class)

########################## Caret Preparation ############################
k.folds = 10
allSummary  <- function(data, lev = NULL, model = NULL){
a1 <- defaultSummary(data, lev, model)
b1 <- twoClassSummary(data, lev, model)
c1 <- prSummary(data, lev, model)
out <- c(a1, b1, c1)
out
# return(out)
}
df1.control <- trainControl( # 10 Fold cross validation, repeated 5 times
method="repeatedcv",
number=k.folds,
repeats=5,
classProbs = T,
summaryFunction=allSummary
#savePredictions=T
)

########################## Hyperparametertuning NeuralNet (MLP) ############################
df1.tunegrid <- expand.grid(.size=c(1:(ncol(df1)-1)))
metric <- "Accuracy"

set.seed(1337)
tic("MLP DF1, Hyperparameter Startegy 1: Grid Search")
mlp_df1 <- train(Class~., data=df1, method="mlp", metric=metric, tuneGrid=df1.tunegrid, trControl=df1.control)
toc()
print(mlp_df1)
# plot(mlp_df1)
print(mlp_df1$bestTune) library("iml") x <- select(df1,-Class) y <- select(df1,Class) predictor <- Predictor$$new(mlp_df1, data = x, y = y, type = "prob") hp_size <- as.numeric(mlp_df1$$bestTune) # Tuned Hyperparameter Neruon in hiddenlayer metric = "ce" # allowed losses "ce", "f1", "logLoss", "mae", imp <- FeatureImp$new(predictor,
loss = metric,
compare = "ratio",
n.repetitions = 50
)
plot(imp)


Edit 2 - Solution for the toy example: ok it was pretty simple after all. The MLP has 100% Accuracy, meaning the error baseline is 0. The default importance score computation via ratio (which is described as example in the IML-Book) would result in an error: divide by 0. So the package switches to compare = "difference". And for this a range between 0 and 1 should be correct. This still leaves me guessing about our dataset, since the MLP there doesn't achieve 100% accuracy.

References
Molnar, C., Casalicchio, G., & Bischl, B. (2018). iml: An R package for Interpretable Machine Learning. Journal of Open Source Software, 3(26), 786.
Fisher, A., Rudin, C., & Dominici, F. (2018). Model Class Reliance: Variable importance measures for any machine learning model class, from the ‘Rashomon’ perspective.