11
$\begingroup$

I came across one comment in an xgboost tutorial. It says "Remember that gamma brings improvement when you want to use shallow (low max_depth) trees".

My understanding is that higher gamma higher regularization. If we have deep (high max_depth) trees, there will be more tendency to overfitting. Why is it the case that gamma can improve performance using shallow trees?

Here is the tutorial link https://www.hackerearth.com/fr/practice/machine-learning/machine-learning-algorithms/beginners-tutorial-on-xgboost-parameter-tuning-r/tutorial/

$\endgroup$
2
  • $\begingroup$ Can you please provide a reference for that tutorial comment? It is hard to contextualise it! (Welcome to the CV community) $\endgroup$
    – usεr11852
    Jul 29 '19 at 23:52
  • $\begingroup$ Thanks! I have added the tutorial link. $\endgroup$ Jul 31 '19 at 17:23
9
$\begingroup$

As you correctly note gamma is a regularisation parameter. In contrast with min_child_weight and max_depth that regularise using "within tree" information, gamma works by regularising using "across trees" information. In particular by observing what is the typical size of loss changes we can adjust gamma appropriately such that we instruct our trees to add nodes only if the associated gain is larger or equal to $\gamma$. In the rather famous 2014 XGBoost presentation by Chen, p.33 refers to $\gamma$ as the "complexity cost by introducing additional leaf".

Now, a typical situation where we would tune gamma is when we use shallow trees as we try to combat over-fitting. The obvious thing to combat overfitting is use shallower trees (i.e. lower max_depth) and therefore the context where tuning gamma becomes relevant is "when you want to use shallow (low max_depth) trees". Indeed it is a bit of a tautology but realistically if we expect deeper trees to be beneficial, tuning gamma, while still effective in regularising, will also unnecessarily burden our learning procedure. On the other hand, if we wrongly use deeper trees, unless we regularise very aggressively we might be accidentally end-up to a local minimum where $\gamma$ cannot save us from. Therefore, $\gamma$ is indeed more relevant for "shallow-tree situations". :) A great blog post on tuning $\gamma$ can be found here: xgboost: “Hi I’m Gamma. What can I do for you?” - and the tuning of regularization.

Final word of caution: Do notice that $\gamma$ is strongly dependant on the actual estimated parameters and the (training) data. That is because the scale of our response variable effectively dictates the scale of our loss function and what are the subsequent reductions in the loss function (i.e. values of $\gamma$) we consider meaningful.

$\endgroup$
5
  • $\begingroup$ I don't think I agree with "if we wrongly use deeper trees, ... gamma cannot save us." If gamma is sufficiently large, it will prevent any splits from happening anywhere, and the max_depth parameter becomes meaningless. $\endgroup$ Jul 31 '19 at 21:12
  • $\begingroup$ In which case we work with stumps and brings us back to the point of gamma being more relevant for shallow trees... :) Anyway, no problem. I will try amend it my words a bit so it is more clear, thanks. $\endgroup$
    – usεr11852
    Jul 31 '19 at 22:10
  • $\begingroup$ You mentioned a local minimum. Is that a known issue with xgboost? Shouldn't the loss functions be convex for binary:logistic or reg:squarederror? Convext w.r.t. the predictions, since this is the grad/hess in function space, rather than in parameter space as in neural networks. $\endgroup$ Jan 19 '21 at 12:04
  • $\begingroup$ I would see it as an issue in almost all GBM algorithms. Off the top of my head I can think three reasons: 1. we work a convex approximation of the true cost. 2. in each step we work with a subsample of the original data, (so even we accept that we reach the global minimum for that iteration that is subject to change), 3. CART trees are greedy algorithms, as such they are not guaranteed to converge to converge to a global minimum themselves. $\endgroup$
    – usεr11852
    Jan 19 '21 at 23:00
  • $\begingroup$ I was pointing out that with popular loss functions like binary:logisticc and reg:squarederror, they actually are convex themselves -- not just the 2nd order approximations. So while subsampling and other things might cause convergence issues, is it quite right to call this a problem with local minima? (I'm not an expert) $\endgroup$ Feb 9 '21 at 13:47
1
$\begingroup$

Gamma causes shallower trees (or, at least, trees with fewer leaves), by restricting when splits will be made.

I think the tutorial isn't entirely clear/accurate. For example, the bullet point immediately before the one you're questioning states that gamma penalizes large coefficients, which is not the case (alpha and lambda penalize coefficients, gamma just penalizes the number of leaves). And further down, alpha is the L1 penalty on weights, but the weights are those at each leaf, not on individual features, so alpha does not perform feature selection in the same way as Lasso. (I suppose, though I haven't seen it discussed, that it could force a split candidate's leaf coefficient to zero, causing the algorithm to pass over splitting that feature, perhaps in the long run skipping the feature altogether?)

$\endgroup$
3
  • $\begingroup$ Please see my post where I tried to explain where the author is coming from. You are right that probably a sentence or two more would be needed to clarify what is going on, but the author's statement in question is not inaccurate. Indeed $\gamma$ is "more relevant" in "shallow tree" use cases. $\endgroup$
    – usεr11852
    Jul 31 '19 at 19:02
  • 1
    $\begingroup$ @usεr11852, I agree that the author is right in the practical advice: increasing gamma increases regularization, combatting overfitting, and so "gamma brings improvement with shallow trees" could maybe be read as "gamma creates shallower trees"; and the general ideas on the other bullet points work out for practice use as well. But it doesn't seem they have a clear grasp on the parameters. I'd be interested to hear whether you think alpha can (or is likely to) prune out features. $\endgroup$ Jul 31 '19 at 21:08
  • $\begingroup$ I find $\alpha$ being relevant if we use the linear booster. It apparently is relevant for the tree learner too but I have not studied it. As far as I can see, alpha will directly affect the loss as large weights will be penalised. In that sense this probably will result in shallower trees as it will encourage sparsity in the leafs. I somehow suspect it acts a bit as an inverted min_child_weight. (Damn you, I had to spend 30' on this comment. Thanks though, I learned something. Probably this will be a good question on its own right.) $\endgroup$
    – usεr11852
    Jul 31 '19 at 22:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.