Suppose there are $5$ sample units, $2$ of which carry the feature $y=1$ to be predicted and three of which carry the feature $y=0$. So, $2$ are positive. The XGBoost algorithm initializes with
$\hat\theta_0=argmin_\theta\sum_{i=1}^nL(y_i,\theta)$,
where $L(y_i,\theta)$ is the logarithmic loss $L(y_i,\theta)=\frac{1}{n}\sum_{i=1}^ny_i\log(p_i)+(1-y_i)\log(1-p_i)$,
where $p_i=\frac{1}{1+e^{-x_i}}.$
Now, I am struggling a bit with how to calculate the initial leaf weights. These should be represented by $x_i$ in the equation as far as I understand? From a common sense perspective, we should start with $\theta_0=0.4$ (which is the sample mean and maximum likelihood estimator of a bernoulli distribution). But how can I solve the minimization problem so that I come to this result?
$L(y_i,\theta)$ solves to $2*log(p_i)+3*log(1-p_i)$. But, this should be solved by $\theta_0=x=1$ which seems to be an odd initial value to start the XGBoost algorithm. Or is XGBoost actually starting with a prediction of 1 for every sample unit?
Okey, so I have an answer to the question: I have to differentiate with respect to $\theta=p_i$ instead of $x_i$. Then, $2*log(p_i)+3*log(1-p_i)$ is minimized by $\theta=p_i=0.4$.
