21
$\begingroup$

I am unable to fully understand how this parameter works from the description in the documentation

[max_delta_step [default=0]] Maximum delta step we allow each tree’s weight estimation to be. If the value is set to 0, it means there is no constraint. If it is set to a positive value, it can help making the update step more conservative. Usually this parameter is not needed, but it might help in logistic regression when class is extremely imbalanced. Set it to value of 1-10 might help control the update

It is not clear to me what the "delta step" refers to, especially since there is already an analytical solution to the weights, and the weights are already penalized by eta. Can someone shed some light on where this parameter fits into the algorithm, what the "delta step" refers to, and how exactly does it help in extremely imbalanced datasets?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
14
$\begingroup$

eta introduces a 'relative' regularization (multiplying the weight by a constant factor) but in extreme cases where hessian is nearly zero (like when we have very unbalanced classes) this isn't enough because the weights (in which computation the hessian is in denominator) becomes to nearly infinite. So what max_delta_steps do is to introduce an 'absolute' regularization capping the weight before apply eta correction.

If you see the code of xgboost (file parameter.h, procedure CalcWeight), you can see this, and you see the effect of other regularization parameters, lambda and alpha (that are equivalents to L1 and L2 regularization). In special lambda effect complement (or may substitute) max_delta_step, as a lambda greater than zero do the weight smaller.

Improve this answer
$\endgroup$
1
  • $\begingroup$ This is more of a question to @Jag on his answer.(cannot comment due to lower points) To answer the original question eta is multiplied by the weight of the tree. max delta step is the maximum allowed weight for each tree, default value 0 means no restriction on the max value of weight, So if weight becomes infinitely large eta wont help in controlling the contribution, as eta*wt will still be large. Please correct if any of the following formulas are wrong. Wt is = -G/(H+lambda).... G is gradient, H is hessian. Default value of lambda(l2) in xgboost is 1, G is prob-y H is prob(1-prob) i $\endgroup$
    – T.singh
    Feb 14, 2021 at 6:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.