Boosted Trees classification

Question

I'm using R's gbm() package to do a boosted classification problem, where my response variable is a binary variable taking values of 0 and 1. I have 11 predictors in my data set. After running the gbm() procedure, I then obtained a plot of the relative importance for each variable. Basically, it says that my variable X2 has a relative influence of 95.5%.

My problem is that I tried to actually confirm that this is true, or rather, visualize the accuracy of this result. I therefore created a scatter plot of X2 vs. the binary response variable, but there is no clear correlation between the 2 variables, despite a very strong relative influence. Perhaps a logistic regression would work, but the pseudo-R^2 values were almost zero.

Does anyone know in this case what a 95.5% relative influence would even mean? Thanks!

Answer 1

Relative influence is a measure of how useful certain variable was for the model; it does not have a simple interpretation like weights in linear regression. Neither it is an indication that the variable is actually important -- as prashanth noted the model could be overfitted, there is also always a question how large influence is actually statistically significant.

About scatterplot; GBM is fitting a pretty complex model, it is not necessary that it consists of a linear relationship between X2 and the decision. It may be more complex or simply involve other variables (read about XOR problem to see how interaction may completely hide individual impact).

Answer 2

Perhaps your model is over-fitting. Before you apply the classification algorithm, you need to divide the dataset to training and testing sets. The performance measures for the training and testing set (accuracy, sensitivity and specificity) must be close.

I therefore created a scatter plot of X2 vs. the binary response variable, but there is no clear correlation between the 2 variables, despite a very strong relative influence.

Perhaps a box plot would be more appropriate to visualize the discriminatory nature of the variable.

Also there are few parameters in GBM which you need to fine tune such as the number of weak learners (possibly decision trees), the learning rate, the tree dept (only if the weak learner is decision tree) etc, which you need to find based on 10-fold cross-validation.