# Peeking Inside the Black Box, can Feature importance indicate overfit?

my basic question is: can permutation feature importance be used to identify overfitting?

when you have a binary classifcation problem with balanced classes (i.e. 70 x yes, 70 x no), when none of the predictors is relevant for the classification task you would expect an accuracy of 50% right?

We have trained an MLP with an Accuarcy of 95%, which indicated that some of the predictors are relevant for the binary classification task. We computed the permutation importance with the IML R-package, and got an permutation importance score (method difference so permutation error - baseline error) of 0.08 for only one relevant predictor. All other predictors had an importance score of 0 (no change when the predictor is shuffled)

Does this mean we are overfitting? Permutation of the only important predictor results in an decrease of accuracy of 8% we are still at an accuracy of 87% (95% baseline accuracy - 8% of that feature).

Edit - Splitting to validate overfit:

After some helpful comments suggested I need to split the data to be sure about overfit I did.
I used 100 observations (72.5% of the Data) for the Training Data in grouped 5-fold CV gave me an accuarcy of 98.04% for the hyperparameter size of 5 (5 Neurons in the single hidden layer).
Predictions based on the final caret model about the unseen left 38 observations (27.5% of the Data) resulted in 100% Accuracy.

However the sum of my permutation feature importance (see below) is still only 6.2%. 3 Predictors are deemed relevant (permutation of them resulted in an increase of error). Yet it is unclear to me how one can interpret this result: Baseline Accuracy 98.04% - 6.2% Error increase for permutation relevant features = still 91.84% when all relevant features are permuted.

Here is my partial code, I am sorry but I cant give out a reproducable example since I dont own the data I am working with.

  # preparation for the contrasting dataset
df_test <- subset(df,  Class == "Control" | Class == classes[i])
df_test$$Class <- factor(df_test$$Class)
preproc <- select(df_test,-ID,-Class) %>% caret::preProcess(., method = c("center","scale","zv"), verbose = T)
df.contrast <- predict(preproc, df_test)
# Splitting: testing for overfit splitting of the contrast ds
partition = 0.725
train_ids = sample(unique(df.contrast$$ID), size=partition*length(unique(df.contrast$$ID)))
df.train = df.contrast %>% filter(ID %in% train_ids)
df.test = df.contrast %>% filter(!ID %in% train_ids)
# train the model and evaluate on test set
model <- tuneModel.contrast(df.train)
x.test <- select(df.test,-ID,-Class)
predictions.testset = caret::predict.train(model, newdata = x.test)
print(confusionMatrix(predictions.testset, df.test$Class)) permutation_vip.perclass(df.contrast,classes[i],levels(df.contrast$Class),model)


Here the function used to train the model

allSummary  <- function(data, lev = NULL, model = NULL){
a1 <- defaultSummary(data, lev, model)
b1 <- twoClassSummary(data, lev, model)
c1 <- prSummary(data, lev, model)
out <- c(a1, b1, c1)
out
# return(out)
}
########################## Hyperparametertuning NeuralNet (MLP) ############################
tuneModel.contrast <- function(contrast_df){
############## Caret Preparation ##############
set.seed(1337)
k.folds = 5
contrast_df.folds <- groupKFold(contrast_df$ID, k = k.folds) contrast_df.control <- trainControl( # k Folds grouped by subject cross validation, repeated 3 times method = "repeatedcv", number = k.folds, repeats = 3, index = contrast_df.folds, savePredictions = T, summaryFunction = allSummary, classProbs = T ) contrast_df <- select(contrast_df, -"ID") contrast_df.tunegrid <- expand.grid(.size=c(1:(ncol(contrast_df)-1))) metric <- "Accuracy" # metric <- "AUC" tic("MLP Contrasting, Hyperparameter Startegy 1: Grid Search") mlp <- train(Class~., data=contrast_df, method="mlp", metric=metric, tuneGrid=contrast_df.tunegrid, trControl=contrast_df.control # , preProc=c("center", "scale","zv") ) toc() print(mlp) # M <- mlp$results
# print(sort(apply(M,2,sd), decreasing = T))
return(mlp)
}

• Seems plausible - have you looked at the more standard ways of checking for overfitting? Applying your fitted model to a test set and checking performance there, for example? – mkt - Reinstate Monica Aug 16 '19 at 9:51
• Thanks for your response. Sadly I only have an N of 138. I think splitting isn't an option with such a small sample right? I've trained the mlp with and grouped 5-fold cross validation using caret, maybe looking at the respective performance for each of the 5 folds could help? – PythonBeginner Aug 16 '19 at 9:54
• Not my area of expertise, unfortunately, so I'll refrain from commenting further. Hopefully you will get more informed answers soon. – mkt - Reinstate Monica Aug 16 '19 at 9:56
• You have to split. You can try leave-k-out CV without any problems on this data set. – Digio Aug 19 '19 at 8:09
• Your main question (title) seems very interesting in general. For your specific problem I think the correct way to go would be some form of regularisation in the model fitting process. – lcrmorin Aug 19 '19 at 9:33