I am attempting to understand interactions in an xgboost model. I have the following reproducible example in R where I create a model with the diamonds dataset and calculate the interaction metrics for all variables along with the 2-way interactions. My question is: how do I determine what constitutes a "high" or "significant" interaction. The EIX package will produce a plot of the interactions Gain score, and color-code them by strength, but does not describe what the cutoff for "high" is.
Furthermore, the Gain of an individual variable when only individual variables are considered is greater than the Gain when interactions are also considered, and I assume this is because the additional Gain from the interactions are included in the totals for the individual variable in the former case. However, the total Gain for all interactions is much larger than the difference in total gain attributed to all individual variables plus interactions minus the total Gain attributed to just individual variables.
I seem to be missing a key piece of understanding. Any help or insights from others with more experience would be much appreciated. Thank you!
library(EIX) library(recipes) library(xgboost) library(caret) library(tidyverse) options(scipen = 9999) d <- data.frame(diamonds) d$cut <- as.character(d$cut) d$color <- as.character(d$color) d$clarity <- as.character(d$clarity) # set up a recipe to one hot encode categorical variables rec <- recipe(price ~ ., data = d) %>% recipes::step_dummy(cut, color, clarity) # prep recipe to make metadata for data preprocessing rec_prep <- prep(rec) # bake recipe to create a new data.frame df_new <- bake(rec_prep, d) # subsample the dataset to make smaller set.seed(5) df_new <- df_new %>% sample_n(2000) # set up data for xgboost df_m <- as.matrix(df_new[, -7]) label <- df_new$price # run an xgb model and get the importance of interactions and plot param <- list(max_depth = 4) xgb_model <- xgboost(df_m, label = label, params = param, nrounds = 50, verbose = TRUE) ints <- EIX::interactions(xgb_model = xgb_model, data = df_m) plot(ints, radar = FALSE) # get the importance of individual variables together with the importance of the interactions importance <- EIX::importance(xgb_model, df_m, option = "both") data.frame(importance$Feature, importance$sumGain) # y has sumGain = 27780000000 total_gain_both <- sum(importance$sumGain) # get the importance of individual variables only importance_vars <- EIX::importance(xgb_model, df_m, option = "variables") data.frame(importance_vars$Feature, importance_vars$sumGain) # gain greater for individual vars here compared to what's listed above # y has sumGain = 27970000000, is this because the sumGain includes the amounts attributed to the interactions above? total_gain <- sum(importance_vars$sumGain) # difference in total gain from both option to variables option total_gain_both - total_gain # 12309340 # get the importance of just the interactions importance_ints <- EIX::importance(xgb_model, df_m, option = "interactions") data.frame(importance_ints$Feature, importance_ints$sumGain) total_gain_ints <- sum(importance_ints$sumGain) total_gain_ints # 1654879940 # this is much larger than total_gain_both - total_gain int_xgb <- xgb.importance(model = xgb_model) # y has Gain 0.4699949981 signif(total_gain*0.4699949981,4) # this is same as sumGain attributed to importance when single variable is calculated from EIX package ```