I tried to use a GBM model to make predictions for the same data point, but it gave me very different answers. Please see the example below. When using the entire dataset for predicting the first data point, the predicted value is 49.30266. However, when I pulled out the first data point alone for prediction (with the correct factor level for treatment), the results were not consistent. Then, I put into the first observation manually, i.e., manualFirstrow0 and manualFirstrow1 (with wrong treatment value), but they produced the same estimation.

Obviously, the package (GBM version 2.1.4) may have some issues when it comes to categorical features. The correct prediction should be around 50 like presented above. Anyone has any ideas how to deal with this issue?

> set.seed(1)
> data(OrchardSprays)
> model <- gbm(decrease ~ rowpos+colpos+treatment, data=OrchardSprays, n.trees=1000, distribution="gaussian", interaction.depth=3, bag.fraction=0.5, train.fraction=1.0, shrinkage=0.1, keep.data=TRUE)
> predict(model,newdata=OrchardSprays,n.trees=100)[1]
[1] 49.30266
> firstrow <- OrchardSprays[1,]
> manualFirstrow0 <- data.frame(decrease=57,rowpos=1,colpos=1,treatment="D")
> manualFirstrow1 <- data.frame(decrease=57,rowpos=1,colpos=1,treatment="A")
> predict(model,newdata=firstrow,n.trees=100)
[1] 16.12713
> predict(model,newdata=manualFirstrow0,n.trees=100)
[1] 16.12713
> predict(model,newdata=manualFirstrow1,n.trees=100)
[1] 16.12713

Here is the link to the original question: Why does GBM predict different values for the same data. But it seems that we cannot solve this issue by fixing the factor issue.


Unfortunately you have stumbled upon a bug; you read more about here at: https://github.com/gbm-developers/gbm/issues/28. The developers are aware of it and they have provided a solution on the development branch of the package gbm.

Notice that this bug relates on how factors are encoded. If we perform the factor encoding manually outside the function gbm we circumvent this issue. The two most obvious encoding schema are the one-hot encoding where we treat the factor variable treatment as an indicator variable directly (thus we end up with $p$ 0/1 variables where $p$ is the number of levels in our factor variable) and the simple numeric encoding where we assign a random integer to each level of the factor variable. So, for example, we could do:

OrchardSpraysNUM <- OrchardSprays[, -which(colnames(OrchardSprays) == "treatment")]
OrchardSpraysNUM$treatment <- as.numeric(OrchardSprays$treatment)

modelN <- gbm(decrease ~ . , data=OrchardSpraysNUM, n.trees=1000,
              distribution="gaussian", interaction.depth=3, bag.fraction=0.50,
              train.fraction=1.0, shrinkage=0.1, keep.data=TRUE)

predict(modelN,newdata=OrchardSpraysNUM, n.trees=100)[1]
# [1] 51.55688

firstrowNUM <- OrchardSpraysNUM[1,]
predict(modelN,newdata=firstrowNUM, n.trees=100)
# [1] 51.55688

It can be argued that exactly because learning a tree classifier's performing recursive is effectively performing regression using piece-wise flat functions, we could "simply" pass a factor variable as numeric especially when the number of factor levels become large. I have read some articles (e.g. here) that have explored this idea further, and I have seen very successful software implementation (e.g. catboost's take on Transforming categorical features to numerical features) take similar approaches at times.

So to summarise, either install the development version of gbm or encode the factors directly.

| cite | improve this answer | |
  • $\begingroup$ BTW, welcome to the CV community! $\endgroup$ – usεr11852 Jan 8 '19 at 23:04
  • $\begingroup$ Thank you for your comprehensive answer and providing two different solutions. Appreciate it! $\endgroup$ – X. Zhao Jan 12 '19 at 22:14
  • $\begingroup$ I am glad I could help! If you find this answer helpful you could consider upvoting it or if it answers your question, accept it as an answer. If you need further clarifications you are welcome to ask. $\endgroup$ – usεr11852 Jan 12 '19 at 22:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.