4
$\begingroup$

I created a model using the gbm() function in library(gbm). Within the gbm() function, I set the distribution as "adaboost". I have a binary response [0, 1]. I used the predict.gbm function for prediction, but the output is not [0, 1], but real numbers that are both negative and positive.

If within my predict.gbm function, I set type = "response", I believe I get the probability of Y | X = 1. Is this correct?

If I do not set type = "response" what are those values? How would I manually convert it to probabilities?

https://cran.r-project.org/web/packages/gbm/gbm.pdf

$\endgroup$

1 Answer 1

3
$\begingroup$

According to gbm's reference manual: While indeed type="response" then gbm converts back to the same scale as the outcome this currently will be returning probabilities for bernoulli and expected counts for poisson only. For all other distributions response and link return the same. That said, gbm.predict will indeed transform the response when the assumed cost function is Adaboost.

Here is a small R example about it:

rm(list=ls())
data(abalone, package = "AppliedPredictiveModeling") 
library(data.table)  
library(gbm)
setDT(abalone)  
K = 3000
set.seed(3)
gbm_ber <- gbm(as.numeric("M"==Type) ~ Diameter + WholeWeight + ShellWeight + Height, 
               data=abalone[1:K,], distribution="bernoulli")
gbm_ada <- gbm(as.numeric("M"==Type) ~ Diameter + WholeWeight + ShellWeight + Height, 
               data=abalone[1:K,], distribution="adaboost")

par(mfrow=c(2,2))
plot(density(predict(gbm_ber, newdata=abalone[-c(1:K),], n.trees=100, type="link")),
     main="Bernoulli - Link")
plot(density(predict(gbm_ber, newdata=abalone[-c(1:K),], n.trees= 100, type="response")),
     main="Bernoulli - Response")
plot(density(predict(gbm_ada, newdata=abalone[-c(1:K),], n.trees=100, type="link")), 
     main="Adaboost - Link")
plot(density(predict(gbm_ada, newdata=abalone[-c(1:K),], n.trees=100, type="response")), 
     main="Adaboost - Response")

enter image description here

As we can see the response predictions are indeed between $[0,1]$ for Adaboost too. The manually conversion to probabilities between the link predictions and response predictions in the case of Bernoulli requires the using the inverse logit: $\frac{1}{1+exp(-x)}$ while for the Adaboost is $\frac{1}{1+exp(-2x)}$. The $2$ comes directly from the inversion of $\frac{1}{2} \ln(\frac{1-\epsilon}{\epsilon})$, see Schapire (2013) "Explaining Adaboost" for more details. We can see the the manual calculations here:

predict(gbm_ada, newdata=abalone[K+c(1:4),], n.trees=100, type="response")
# [1] 0.4851037 0.4913199 0.4791932 0.5135091
predict(gbm_ber, newdata=abalone[K+c(1:4),], n.trees=100, type="response")
# [1] 0.4730339 0.4943825 0.5105203 0.5063024

1/(1+exp(-2*predict(gbm_ada, newdata=abalone[K+c(1:4),], n.trees=100, type="link")))
# [1] 0.4851037 0.4913199 0.4791932 0.5135091
1/(1+exp(-1*predict(gbm_ber, newdata=abalone[K+c(1:4),], n.trees=100, type="link")))
# [1] 0.4730339 0.4943825 0.5105203 0.5063024
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.