1
$\begingroup$

One way to explain individual predictions of an xgb classifier is to calculate contributions of each feature. To my knowledge there are two packages in R that can do this for you automatically. In the xgboost package you can call the predict.xgb.Booster function at set predcontrib to TRUE. In the xgboostExplainer package you can call the buildExplainer and explainPredictions functions.

Both methods result in a n-by-m dataframe where n = the number of observations and m = the number of features. However, in my hands the two methods give different values for the contributions of each prediction. These differences exceed floating point error, so it suggests different calculations are being performed. Interestingly, the difference in the final prediction values (sum of prediction contributions) is within floating point error.

Two specific questions:

  • Is there a preferred method within the stats community for calculating the contribution of each feature in an xgb classifier?

  • Why the discrepancy between the feature contributions between the two methods?

Reproducible example follows:

library(xgboost)
library(xgboostExplainer)

## binary classification:

data(agaricus.train, package='xgboost')
data(agaricus.test, package='xgboost')
train <- agaricus.train
test <- agaricus.test

bst <- xgboost(data = train$data, label = train$label, max_depth = 2,
               eta = 0.5, nthread = 2, nrounds = 5, objective = "binary:logistic")

pred <- predict(bst, test$data)
pred_contr <- predict(bst, test$data, predcontrib = TRUE)
contr1 <- pred_contr[1,]

xgb.train.data <- xgb.DMatrix(train$data, label = train$label)
xgb.test.data <- xgb.DMatrix(test$data, label = test$label)
explainer = buildExplainer(bst, xgb.train.data, type="binary", base_score = 0.5, trees = NULL)

pred.breakdown = explainPredictions(bst, explainer, xgb.test.data)

#discrepancy between prediction contributions

breakdown1 <- pred.breakdown[1,]

sum(contr1) - sum(breakdown1)

summary(as.numeric(contr1) - as.numeric(breakdown1[1,]))


## Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
#-0.2982  0.0000  0.0000  0.0000  0.0000  0.1712 

#but not between final predictions

summary(rowSums(pred_contr) - rowSums(pred.breakdown))

#      Min.    1st Qu.     Median       Mean    3rd Qu.       Max. 
##-1.612e-07 -2.644e-08 -1.401e-08  2.783e-08  6.149e-08  3.226e-07 
$\endgroup$
1
$\begingroup$

After digging through the code, I think its safe to say that using predict and setting predcontrib to TRUE is preferable to using the xgboostExplainer functions.

The following code demonstrates that the discrepancy between the two packages disappears when you set approxcontrib to TRUE.

library(xgboost)
library(xgboostExplainer)

## binary classification:

data(agaricus.train, package='xgboost')
data(agaricus.test, package='xgboost')
train <- agaricus.train
test <- agaricus.test

bst <- xgboost(data = train$data, label = train$label, max_depth = 2,
               eta = 0.5, nthread = 2, nrounds = 5, objective = "binary:logistic")

pred <- predict(bst, test$data)
pred_contr <- predict(bst, test$data, predcontrib = TRUE, approxcontrib = FALSE)
pred_approx_contr <- predict(bst, test$data, predcontrib = TRUE, approxcontrib = TRUE)

xgb.train.data <- xgb.DMatrix(train$data, label = train$label)
xgb.test.data <- xgb.DMatrix(test$data, label = test$label)
explainer = buildExplainer(bst, xgb.train.data, type="binary", base_score = 0.5, trees = NULL)

pred_xgboostExplainer = explainPredictions(bst, explainer, xgb.test.data)

#similarly shaped outputs
dim(pred_xgboostExplainer)
dim(pred_contr) 
dim(pred_approx_contr)

#discrepancy between prediction contributions when approxcontrib = FALSE

summary(as.numeric(pred_contr[1,]) - as.numeric(pred_xgboostExplainer[1,]))

#    Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
# -0.2982  0.0000  0.0000  0.0000  0.0000  0.1712 

#discrepancy disappears when approxcontrib = TRUE

summary(as.numeric(pred_approx_contr[1,]) - as.numeric(pred_xgboostExplainer[1,]))

#       Min.    1st Qu.     Median       Mean    3rd Qu.       Max. 
# -4.525e-08  0.000e+00  0.000e+00  3.522e-10  0.000e+00  6.590e-08 

#final prediction values all agree with output of predict function

summary(qlogis(pred) - rowSums(pred_xgboostExplainer)) # Max ~ 1e-07
summary(qlogis(pred) - rowSums(pred_approx_contr)) # Max ~ 1e-07
summary(qlogis(pred) - rowSums(pred_contr)) # Max ~ 1e-07

So now I think I can answer the two questions posed above.

Why the discrepancy between the two methods?

The approximation is the "path" method developed in this blog post and was likely the inspiration for the xgboostExplainer blog post. Lundberg et al note that the path method just takes the single ordering defined by descending the tree, whereas their method (which is what is used in the xgboost package) is the average of the path method computed over all permutations of possible variable orderings. By only considering a single ordering, the path method runs into inconsistency problems as shown in Fig 1 of Lundberg et al.

Is there a preferred method when calculating the contribution of each feature in an xgb classifier?

Because predict can calculate everything that xgboostExplainer can and more, I think its safe to say that predict should be preferred. It is 100x faster and available natively in the xgboost package, whereas xboostExplainer is not available on CRAN. Also you should probably set approxcontrib to FALSE unless you have a good reason not to.

(That said, I still like the waterfall plotting functions in xgboostExplainer.)

$\endgroup$
  • 1
    $\begingroup$ +1 Nice answer and good on you for posting the answer back. (I had noted your question to revisit it during the weekend.) You might want to explore the use of package iml to get more information about your model. $\endgroup$ – usεr11852 says Reinstate Monic Oct 10 '18 at 21:25
  • $\begingroup$ @usεr11852 thanks! I'll leave my answer unselected for awhile in case folks find they want to add their two cents. $\endgroup$ – Alexander Reeves Oct 11 '18 at 2:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.