How do you interpret prediction output in GBM() in R for classification problem? I created a model using the gbm() function in library(gbm).  Within the gbm() function, I set the distribution as "adaboost".  I have a binary response [0, 1].  I used the predict.gbm function for prediction, but the output is not [0, 1], but real numbers that are both negative and positive.
If within my predict.gbm function, I set type = "response", I believe I get the probability of Y | X = 1.  Is this correct?
If I do not set type = "response" what are those values?  How would I manually convert it to probabilities?
https://cran.r-project.org/web/packages/gbm/gbm.pdf
 A: According to gbm's reference manual: While indeed type="response" then gbm converts back to the same scale as the outcome this currently will be returning probabilities for bernoulli and expected counts for poisson only. For all other distributions response and link return the same.
That said, gbm.predict will indeed transform the response when the assumed cost function is Adaboost. 
Here is a small R example about it:
rm(list=ls())
data(abalone, package = "AppliedPredictiveModeling") 
library(data.table)  
library(gbm)
setDT(abalone)  
K = 3000
set.seed(3)
gbm_ber <- gbm(as.numeric("M"==Type) ~ Diameter + WholeWeight + ShellWeight + Height, 
               data=abalone[1:K,], distribution="bernoulli")
gbm_ada <- gbm(as.numeric("M"==Type) ~ Diameter + WholeWeight + ShellWeight + Height, 
               data=abalone[1:K,], distribution="adaboost")

par(mfrow=c(2,2))
plot(density(predict(gbm_ber, newdata=abalone[-c(1:K),], n.trees=100, type="link")),
     main="Bernoulli - Link")
plot(density(predict(gbm_ber, newdata=abalone[-c(1:K),], n.trees= 100, type="response")),
     main="Bernoulli - Response")
plot(density(predict(gbm_ada, newdata=abalone[-c(1:K),], n.trees=100, type="link")), 
     main="Adaboost - Link")
plot(density(predict(gbm_ada, newdata=abalone[-c(1:K),], n.trees=100, type="response")), 
     main="Adaboost - Response")


As we can see the response predictions are indeed between $[0,1]$ for Adaboost too. The manually conversion to probabilities between the link predictions and response predictions in the case of Bernoulli requires the using the inverse logit: $\frac{1}{1+exp(-x)}$ while for the Adaboost is  $\frac{1}{1+exp(-2x)}$. The $2$ comes directly from the inversion of $\frac{1}{2} \ln(\frac{1-\epsilon}{\epsilon})$, see Schapire (2013) "Explaining Adaboost" for more details. 
We can see the the manual calculations here:
predict(gbm_ada, newdata=abalone[K+c(1:4),], n.trees=100, type="response")
# [1] 0.4851037 0.4913199 0.4791932 0.5135091
predict(gbm_ber, newdata=abalone[K+c(1:4),], n.trees=100, type="response")
# [1] 0.4730339 0.4943825 0.5105203 0.5063024

1/(1+exp(-2*predict(gbm_ada, newdata=abalone[K+c(1:4),], n.trees=100, type="link")))
# [1] 0.4851037 0.4913199 0.4791932 0.5135091
1/(1+exp(-1*predict(gbm_ber, newdata=abalone[K+c(1:4),], n.trees=100, type="link")))
# [1] 0.4730339 0.4943825 0.5105203 0.5063024

