5
$\begingroup$

I am reading through Chen's XGBoost paper. He writes that during the $\text{t}^{\text{th}}$ iteration, the objective function below is minimised.

$$ L^{(t)} = \sum_{i}^n l(y_i, \hat{y}_i^{(t-1)} + f_t(x_i)) + \Omega (f_t)$$

Here, $l$ is a differentiable convex loss function, $f_t$ represents the $\text{t}^{\text{th}}$ tree and $\hat{y}_i^{(t-1)}$ represents the prediction of the $\text{i}^{\text{th}}$ instance at iteration $t-1$.

I was wondering what $l$ is when using XGBoost for binary classification?

$\endgroup$
  • 3
    $\begingroup$ $l$ is the log likelihood of the bernoulli distribution. $\endgroup$ – Matthew Drury Apr 24 '18 at 19:41
  • 1
    $\begingroup$ That's correct. The expression you cite takes the role of the linear predictor in logistic regression. $\endgroup$ – Matthew Drury Apr 24 '18 at 19:53
  • 1
    $\begingroup$ The second one, the first one is a log-odds. $\endgroup$ – Matthew Drury Apr 24 '18 at 19:57
  • 1
    $\begingroup$ Yup. You usually rewrite that in a slightly different, but algebraically equivelent, way, as $l = y_i (\hat y_i^{t-1} + f_t(x_i)) - \log(1 + \exp (\hat y_i^{t-1} + f_t(x_i)))$ $\endgroup$ – Matthew Drury Apr 24 '18 at 20:08
  • 1
    $\begingroup$ No problem. I'll try to write this up in an answer tonight so people don't have to dig through the comments to follow the discussion. $\endgroup$ – Matthew Drury Apr 24 '18 at 20:45
5
$\begingroup$

It appears there is an option objective: "binary:logistic"

“binary:logistic” –logistic regression for binary classification, output probability

“binary:logitraw” –logistic regression for binary classification, output score before logistic transformation

See http://xgboost.readthedocs.io/en/latest/parameter.html

(so log loss)

| cite | improve this answer | |
$\endgroup$
0
$\begingroup$

I'm just summarizing the comments under the problem description:

$l$ is the log likelihood of the bernoulli distribution, where $\text{logistic}(\hat{y}_i^{(t-1)} + f_t(x_i))$ is the probability. So the formula should be $$l = y_i \log(\text{logistic}(\hat{y}_i^{(t-1)} + f_t(x_i))) + (1-y_i)\log(1-\text{logistic}(\hat{y}_i^{(t-1)} + f_t(x_i)))$$

or algebraically equivelent, $$l = y_i (\hat y_i^{t-1} + f_t(x_i)) - \log(1 + \exp (\hat y_i^{t-1} + f_t(x_i)))$$ using the property that $\sigma(-z)=1 - \sigma(z)$.

| cite | improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.