Suppose there are $5$ sample units, $2$ of which carry the feature $y=1$ to be predicted and three of which carry the feature $y=0$. So, $2$ are positive. The XGBoost algorithm initializes with


where $L(y_i,\theta)$ is the logarithmic loss $L(y_i,\theta)=\frac{1}{n}\sum_{i=1}^ny_i\log(p_i)+(1-y_i)\log(1-p_i)$,

where $p_i=\frac{1}{1+e^{-x_i}}.$

Now, I am struggling a bit with how to calculate the initial leaf weights. These should be represented by $x_i$ in the equation as far as I understand? From a common sense perspective, we should start with $\theta_0=0.4$ (which is the sample mean and maximum likelihood estimator of a bernoulli distribution). But how can I solve the minimization problem so that I come to this result?

$L(y_i,\theta)$ solves to $2*log(p_i)+3*log(1-p_i)$. But, this should be solved by $\theta_0=x=1$ which seems to be an odd initial value to start the XGBoost algorithm. Or is XGBoost actually starting with a prediction of 1 for every sample unit?

Okey, so I have an answer to the question: I have to differentiate with respect to $\theta=p_i$ instead of $x_i$. Then, $2*log(p_i)+3*log(1-p_i)$ is minimized by $\theta=p_i=0.4$.

  • $\begingroup$ Just to be clear: there are 5 samples and 2 of them are positive? $\endgroup$ – usεr11852 says Reinstate Monic Feb 25 '19 at 23:27
  • $\begingroup$ Congratulations on answering your own question and good on you for sharing the answer! (I had on the back of my head to write a quick answer about it this weekend) You might want write a short concise answer about it so people can potentially upvote it (I would). $\endgroup$ – usεr11852 says Reinstate Monic Mar 1 '19 at 0:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.