0
$\begingroup$

I have build a model with a Gradient Boosting Machine (GBM) and calculated the feature importance. All features are factors.

enter image description here

Now, I know which features are most important. However, the features have different levels. For example, "dinner" has the levels "true"/"false" where NoiseLevel has the levels "high/average/low". How can I find out which factor level of a given variable is most important?

The data is about restaurants. The goal is to predict the review counts by the attributes of a restaurant. I want to know which attributes a restaurant should offer to get as many review counts as possible i.e. a high demand. So "dinner" alone is not very informative.

I also tried dummy variables. Therefore, every factor level has its own column (e.g. the column "dinner" became "diner-true" and "dinner-false" with the values 0 and 1). However, in this case, I got weird results. For example, "dinner-true" and "dinner-false" was almost equally important in the analyses.

Any Idea how to deal with this problem?

Edit:

Here is an example of LIME. The goal is to predict "Sepal.Length" by all other features of the iris data set.

library(lime)
library(mlr)
library(tidyverse)

#Train the model
iris.task = makeRegrTask(data = iris, target = "Sepal.Length")

lrn = makeLearner("regr.gbm", n.trees = 100)

mod = train(lrn, iris.task)

#Evaluate the model
rdesc = makeResampleDesc("CV",
                         iters = 5) 
set.seed(3)
bmr = benchmark(lrn, iris.task, rdesc, measures = list(rmse, rsq))

bmr$results$iris$regr.gbm$aggr
rmse.test.rmse  rsq.test.mean 
 0.3535742      0.8168503

#Calculate LIME
explainer = lime(iris, mod, quantile_bins = FALSE)

explanation = explain(iris[1,], explainer,
                      n_permutations = 5000,
                      n_features = 6)

plot_features(explanation, ncol = 1)

enter image description here

For case 1, a have an R2 of 0.54. If I understand the concept correctly case 1 is representative for the whole model through the permutation. So Petal.Length is the most important feature. My question is why is R2 so different from the cross-validation results? Furthermore, I don´t understand why the target variable is in the explanation. This makes no sense to me.

Is there something I missed?

$\endgroup$
  • $\begingroup$ This is very unclear. What's a gbm? What's a variable shaping? What is your dependent variable? What do you mean by "can be split by the gbm"? $\endgroup$ – Peter Flom Mar 22 at 13:34
  • $\begingroup$ Is it OK this way? Do you need additional information? $\endgroup$ – Banjo Mar 22 at 13:55
  • 3
    $\begingroup$ "I want to know which attributes a restaurant should offer to get as many review counts as possible i.e. a high demand." The "feature importance" measures in tree based models are not designed to answer this question. So, even given the dummy variable issue, using the output of the feature importance metrics is already an issue. You want to study something like partial dependence plots, or use a technique like LIME or SHAP, which directly addresses the question you have. $\endgroup$ – Matthew Drury Mar 22 at 15:06
  • $\begingroup$ @MatthewDrury gave the correct answer. This is a typical case of using PDPS, LIME or SHAP. You need directionality in the effects shown; ie. you need to model interpretability. $\endgroup$ – usεr11852 Mar 22 at 22:35
  • 1
    $\begingroup$ When you created dummy variables did you create $p-1$ variables for each of your $p$ categories? For example, true/false variables only need a single variable. A variable with 3 categories only needs 2 dummies. Otherwise, you have linearly dependent variables which could yield "weird" results as you saw (e.g. Dinner True=1 is equal to Dinner False+1). That being said, you should take the advice provided by Matthew Drury. $\endgroup$ – StatsStudent Mar 24 at 16:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.