1
$\begingroup$

I am attempting to understand interactions in an xgboost model. I have the following reproducible example in R where I create a model with the diamonds dataset and calculate the interaction metrics for all variables along with the 2-way interactions. My question is: how do I determine what constitutes a "high" or "significant" interaction. The EIX package will produce a plot of the interactions Gain score, and color-code them by strength, but does not describe what the cutoff for "high" is.

Furthermore, the Gain of an individual variable when only individual variables are considered is greater than the Gain when interactions are also considered, and I assume this is because the additional Gain from the interactions are included in the totals for the individual variable in the former case. However, the total Gain for all interactions is much larger than the difference in total gain attributed to all individual variables plus interactions minus the total Gain attributed to just individual variables.

I seem to be missing a key piece of understanding. Any help or insights from others with more experience would be much appreciated. Thank you!

library(EIX)
library(recipes)
library(xgboost)
library(caret)
library(tidyverse)

options(scipen = 9999)

d <- data.frame(diamonds)
d$cut <- as.character(d$cut)
d$color <- as.character(d$color)
d$clarity <- as.character(d$clarity)

# set up a recipe to one hot encode categorical variables
rec <- recipe(price ~ ., data = d) %>% recipes::step_dummy(cut, color, clarity)

# prep recipe to make metadata for data preprocessing
rec_prep <- prep(rec)

# bake recipe to create a new data.frame
df_new <- bake(rec_prep, d)

# subsample the dataset to make smaller
set.seed(5)
df_new <- df_new %>% sample_n(2000)

# set up data for xgboost
df_m <- as.matrix(df_new[, -7])
label <- df_new$price

# run an xgb model and get the importance of interactions and plot
param <- list(max_depth = 4)
xgb_model <- xgboost(df_m, label = label, params = param, nrounds = 50, verbose = TRUE)

ints <- EIX::interactions(xgb_model = xgb_model, data = df_m)
plot(ints, radar = FALSE)

# get the importance of individual variables together with the importance of the interactions
importance <- EIX::importance(xgb_model, df_m, option = "both") 
data.frame(importance$Feature, importance$sumGain)
# y has sumGain = 27780000000
total_gain_both <- sum(importance$sumGain) 

# get the importance of individual variables only
importance_vars <- EIX::importance(xgb_model, df_m, option = "variables") 
data.frame(importance_vars$Feature, importance_vars$sumGain) # gain greater for individual vars here compared to what's listed above
# y has sumGain = 27970000000, is this because the sumGain includes the amounts attributed to the interactions above?
total_gain <- sum(importance_vars$sumGain)

# difference in total gain from both option to variables option
total_gain_both -  total_gain
#[1] 12309340

# get the importance of just the interactions
importance_ints <- EIX::importance(xgb_model, df_m, option = "interactions")
data.frame(importance_ints$Feature, importance_ints$sumGain) 
total_gain_ints <- sum(importance_ints$sumGain)
total_gain_ints
#[1] 1654879940 # this is much larger than total_gain_both - total_gain

int_xgb <- xgb.importance(model = xgb_model)
# y has Gain 0.4699949981
signif(total_gain*0.4699949981,4) # this is same as sumGain attributed to importance when single variable is calculated from EIX package
```
$\endgroup$

1 Answer 1

1
$\begingroup$

The interactions plot takes the whole scope of interactions' sumGain [min sumGain,max sumGain] and it divides into four equal intervals - very low, low, medium, high.

Regarding the second issue: Model xgboost creates new tree sequentially node by node and it creates a new node using a variable which has the highest Gain in given moment. Interactions occur when Gain of child is higher than the Gain of parent and as interaction Gain is taken Gain of child variable. Hence, total_gain_both should be equal total_gain.

The difference between total_gain_both and total_gain is caused by approximations of values sumGain for a given variable or interaction (the signif function is used there).

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.