What is the "binary:logistic" objective function in XGBoost?

Question

I am reading through Chen's XGBoost paper. He writes that during the $\text{t}^{\text{th}}$ iteration, the objective function below is minimised.

$$ L^{(t)} = \sum_{i}^n l(y_i, \hat{y}_i^{(t-1)} + f_t(x_i)) + \Omega (f_t)$$

Here, $l$ is a differentiable convex loss function, $f_t$ represents the $\text{t}^{\text{th}}$ tree and $\hat{y}_i^{(t-1)}$ represents the prediction of the $\text{i}^{\text{th}}$ instance at iteration $t-1$.

I was wondering what $l$ is when using XGBoost for binary classification?

Answer 1

It appears there is an option objective: "binary:logistic"

“binary:logistic” –logistic regression for binary classification, output probability

“binary:logitraw” –logistic regression for binary classification, output score before logistic transformation

See http://xgboost.readthedocs.io/en/latest/parameter.html

(so log loss)

Answer 2

I'm just summarizing the comments under the problem description:

$l$ is the log likelihood of the bernoulli distribution, where $\text{logistic}(\hat{y}_i^{(t-1)} + f_t(x_i))$ is the probability. So the formula should be $$l = y_i \log(\text{logistic}(\hat{y}_i^{(t-1)} + f_t(x_i))) + (1-y_i)\log(1-\text{logistic}(\hat{y}_i^{(t-1)} + f_t(x_i)))$$

or algebraically equivelent, $$l = y_i (\hat y_i^{t-1} + f_t(x_i)) - \log(1 + \exp (\hat y_i^{t-1} + f_t(x_i)))$$ using the property that $\sigma(-z)=1 - \sigma(z)$.