0
$\begingroup$

I am trying to improve my understanding of how gbm works and how the model predicts out of sample.

More specifically, suppose you have a classification problem (two outcomes, 0 and 1). When you run a gbm model on a training set, observations are designated a predicted probability based on their characteristics. These predicted probabilities can be extracted quite easily.

Now, when you introduce out-of-sample data, how does gbm attempt to classify observations? Does it assign a predicted probability to observations based on similar values observed in the training set? If so, does gbm use a cutoff value? (i.e. predicted_probability>.5 implies a 1?) I am not sure if that's what gbm does since I cannot seem to find anywhere how to obtain these probabilities.

My goal is to construct an ROC curve from the predictions in the TEST SET, not on the training set, so if anyone has experience/insights about this would be great help

$\endgroup$
1
$\begingroup$

This isn't a question specific to GBMs, but a general machine learning question. In a classification task, most algorithms take in feature values and output what's called a "posterior probability", a number from 0 to 1 indicating the probability of being in the positive class based on your training data.

(The bolded part is actually important, because your training data might not always accurately reflect the distribution of the real problem space, e.g., in unbalanced datasets).

The choice of cutoff for this posterior probability is entirely up to you. In the construction of the ROC curve, you essentially sample cutoffs to obtain different points trading off true positive rate and false positive rate.

Finally, also note that GBMs are a specific algorithm that uses gradient boosting and are not equivalent terms. Gradient boosting is a much more general idea implemented in many different algorithms.

$\endgroup$
  • $\begingroup$ It's worth noting that there are many applications of models where hard classification is not even needed. For many tasks, the conditional probabilities of class membership is sufficient. $\endgroup$ – Matthew Drury Mar 31 '17 at 20:50
  • $\begingroup$ Thank you thc for the explanation. Also, appreciate the clarification on GBM vs gradient boosting. Definitely have some more reading to do. $\endgroup$ – User1414 Apr 1 '17 at 3:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.