10
$\begingroup$

I am reading through Chen's XGBoost paper. He writes that during the $\text{t}^{\text{th}}$ iteration, the objective function below is minimised.

$$ L^{(t)} = \sum_{i}^n l(y_i, \hat{y}_i^{(t-1)} + f_t(x_i)) + \Omega (f_t)$$

Here, $l$ is a differentiable convex loss function, $f_t$ represents the $\text{t}^{\text{th}}$ tree and $\hat{y}_i^{(t-1)}$ represents the prediction of the $\text{i}^{\text{th}}$ instance at iteration $t-1$.

I was wondering what $l$ is when using XGBoost for binary classification?

$\endgroup$
8
  • 3
    $\begingroup$ $l$ is the log likelihood of the bernoulli distribution. $\endgroup$ Commented Apr 24, 2018 at 19:41
  • 1
    $\begingroup$ That's correct. The expression you cite takes the role of the linear predictor in logistic regression. $\endgroup$ Commented Apr 24, 2018 at 19:53
  • 1
    $\begingroup$ The second one, the first one is a log-odds. $\endgroup$ Commented Apr 24, 2018 at 19:57
  • 1
    $\begingroup$ Yup. You usually rewrite that in a slightly different, but algebraically equivelent, way, as $l = y_i (\hat y_i^{t-1} + f_t(x_i)) - \log(1 + \exp (\hat y_i^{t-1} + f_t(x_i)))$ $\endgroup$ Commented Apr 24, 2018 at 20:08
  • 1
    $\begingroup$ No problem. I'll try to write this up in an answer tonight so people don't have to dig through the comments to follow the discussion. $\endgroup$ Commented Apr 24, 2018 at 20:45

2 Answers 2

8
$\begingroup$

It appears there is an option objective: "binary:logistic"

“binary:logistic” –logistic regression for binary classification, output probability

“binary:logitraw” –logistic regression for binary classification, output score before logistic transformation

See http://xgboost.readthedocs.io/en/latest/parameter.html

(so log loss)

$\endgroup$
3
$\begingroup$

I'm just summarizing the comments under the problem description:

$l$ is the log likelihood of the bernoulli distribution, where $\text{logistic}(\hat{y}_i^{(t-1)} + f_t(x_i))$ is the probability. So the formula should be $$l = y_i \log(\text{logistic}(\hat{y}_i^{(t-1)} + f_t(x_i))) + (1-y_i)\log(1-\text{logistic}(\hat{y}_i^{(t-1)} + f_t(x_i)))$$

or algebraically equivelent, $$l = y_i (\hat y_i^{t-1} + f_t(x_i)) - \log(1 + \exp (\hat y_i^{t-1} + f_t(x_i)))$$ using the property that $\sigma(-z)=1 - \sigma(z)$.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.