# What is the “binary:logistic” objective function in XGBoost?

I am reading through Chen's XGBoost paper. He writes that during the $\text{t}^{\text{th}}$ iteration, the objective function below is minimised.

$$L^{(t)} = \sum_{i}^n l(y_i, \hat{y}_i^{(t-1)} + f_t(x_i)) + \Omega (f_t)$$

Here, $l$ is a differentiable convex loss function, $f_t$ represents the $\text{t}^{\text{th}}$ tree and $\hat{y}_i^{(t-1)}$ represents the prediction of the $\text{i}^{\text{th}}$ instance at iteration $t-1$.

I was wondering what $l$ is when using XGBoost for binary classification?

• $l$ is the log likelihood of the bernoulli distribution. – Matthew Drury Apr 24 '18 at 19:41
• That's correct. The expression you cite takes the role of the linear predictor in logistic regression. – Matthew Drury Apr 24 '18 at 19:53
• The second one, the first one is a log-odds. – Matthew Drury Apr 24 '18 at 19:57
• Yup. You usually rewrite that in a slightly different, but algebraically equivelent, way, as $l = y_i (\hat y_i^{t-1} + f_t(x_i)) - \log(1 + \exp (\hat y_i^{t-1} + f_t(x_i)))$ – Matthew Drury Apr 24 '18 at 20:08
• No problem. I'll try to write this up in an answer tonight so people don't have to dig through the comments to follow the discussion. – Matthew Drury Apr 24 '18 at 20:45

It appears there is an option objective: "binary:logistic"

“binary:logistic” –logistic regression for binary classification, output probability

“binary:logitraw” –logistic regression for binary classification, output score before logistic transformation

(so log loss)

I'm just summarizing the comments under the problem description:

$$l$$ is the log likelihood of the bernoulli distribution, where $$\text{logistic}(\hat{y}_i^{(t-1)} + f_t(x_i))$$ is the probability. So the formula should be $$l = y_i \log(\text{logistic}(\hat{y}_i^{(t-1)} + f_t(x_i))) + (1-y_i)\log(1-\text{logistic}(\hat{y}_i^{(t-1)} + f_t(x_i)))$$

or algebraically equivelent, $$l = y_i (\hat y_i^{t-1} + f_t(x_i)) - \log(1 + \exp (\hat y_i^{t-1} + f_t(x_i)))$$ using the property that $$\sigma(-z)=1 - \sigma(z)$$.