0
$\begingroup$

I am aware how GB classifiers are constructed as regression trees and predictions are made, but not sure how the initial tree and node splitting for it is done.

Can someone please explain how the first tree in GBM (classifier) is constructed, and how the node splitting criteria for the first tree is decided. I am not sure, what we are predicting for the first tree (even assuming a constant is initialized, how gradient of loss are computed from the constant).

If mse is the node split criteria, what is it composed of (I mean, squared difference of what values??). Thank you.

$\endgroup$
7
  • 1
    $\begingroup$ I have answered this in the question in the thread: First Iteration in Gradient Boosting Algorithm. If there something more specific you need, please do tell! :) $\endgroup$
    – usεr11852
    Commented Feb 29, 2020 at 9:45
  • $\begingroup$ @usεr11852, thank you for the response. The explanations show how it works for predicting a continuous value ( regression problem), and I understand even in classification we predict the log odds (and hence a regression problem). But in case of classification, wondering how the initial constant before the first iteration is assigned and what is the constant assigned to (log odds??). If the initial constant is assigned to log odds, how do we know, what the true log odds (for true y) should be to calculate the loss-because we only know the true class - say 0 or 1 of sample and not true log odds. $\endgroup$
    – tjt
    Commented Mar 1, 2020 at 16:57
  • 1
    $\begingroup$ This is a bit implementation specific. In most new implementations the exact value of the initial constant depends on the learning rate $\eta$ and/or we can go ahead and directly set it ourselves. Usually we would "boost from average" in the sense that we would calculate the sample log odds despite not being "the true log odds". Assuming that we optimise appropriately, the values of the first iteration are not of great importance. It might affect convergence speed in some cases but realistically if we are prepared to do hundreds of iteration the actual initial values are not very influential. $\endgroup$
    – usεr11852
    Commented Mar 1, 2020 at 19:46
  • 1
    $\begingroup$ I am in full agreement with the answer linked. I am just pointing out the the first estimate of $p_i$ will be the average value observed. We would simply apply the inverse link to it and get $L_i$. $\endgroup$
    – usεr11852
    Commented Mar 2, 2020 at 0:25
  • 1
    $\begingroup$ Cool. I will check the new question within the week. $\endgroup$
    – usεr11852
    Commented Mar 2, 2020 at 0:29

0

Browse other questions tagged or ask your own question.