6
$\begingroup$

I came across one comment in an xgboost tutorial. It says "Remember that gamma brings improvement when you want to use shallow (low max_depth) trees".

My understanding is that higher gamma higher regularization. If we have deep (high max_depth) trees, there will be more tendency to overfitting. Why is it the case that gamma can improve performance using shallow trees?

Here is the tutorial link https://www.hackerearth.com/fr/practice/machine-learning/machine-learning-algorithms/beginners-tutorial-on-xgboost-parameter-tuning-r/tutorial/

$\endgroup$
  • $\begingroup$ Can you please provide a reference for that tutorial comment? It is hard to contextualise it! (Welcome to the CV community) $\endgroup$ – usεr11852 says Reinstate Monic Jul 29 at 23:52
  • $\begingroup$ Thanks! I have added the tutorial link. $\endgroup$ – Salty Gold Fish Jul 31 at 17:23
1
$\begingroup$

As you correctly note gamma is a regularisation parameter. In contrast with min_child_weight and max_depth that regularise using "within tree" information, gamma works by regularising using "across trees" information. In particular by observing what is the typical size of loss changes we can adjust gamma appropriately such that we instruct our trees to add nodes only if the associated gain is larger or equal to $\gamma$. In the rather famous 2014 XGBoost presentation by Chen, p.33 refers to $\gamma$ as the "complexity cost by introducing additional leaf".

Now, a typical situation where we would tune gamma is when we use shallow trees as we try to combat over-fitting. The obvious thing to combat overfitting is use shallower trees (i.e. lower max_depth) and therefore the context where tuning gamma becomes relevant is "when you want to use shallow (low max_depth) trees". Indeed it is a bit of a tautology but realistically if we expect deeper trees to be beneficial, tuning gamma, while still effective in regularising, will also unnecessarily burden our learning procedure. On the other hand, if we wrongly use deeper trees, unless we regularise very aggressively we might be accidentally end-up to a local minimum where $\gamma$ cannot save us from. Therefore, $\gamma$ is indeed more relevant for "shallow-tree situations". :) A great blog post on tuning $\gamma$ can be found here: xgboost: “Hi I’m Gamma. What can I do for you?” - and the tuning of regularization.

Final word of caution: Do notice that $\gamma$ is strongly dependant on the actual estimated parameters and the (training) data. That is because the scale of our response variable effectively dictates the scale of our loss function and what are the subsequent reductions in the loss function (i.e. values of $\gamma$) we consider meaningful.

$\endgroup$
  • $\begingroup$ I don't think I agree with "if we wrongly use deeper trees, ... gamma cannot save us." If gamma is sufficiently large, it will prevent any splits from happening anywhere, and the max_depth parameter becomes meaningless. $\endgroup$ – Ben Reiniger Jul 31 at 21:12
  • $\begingroup$ In which case we work with stumps and brings us back to the point of gamma being more relevant for shallow trees... :) Anyway, no problem. I will try amend it my words a bit so it is more clear, thanks. $\endgroup$ – usεr11852 says Reinstate Monic Jul 31 at 22:10
0
$\begingroup$

Gamma causes shallower trees (or, at least, trees with fewer leaves), by restricting when splits will be made.

I think the tutorial isn't entirely clear/accurate. For example, the bullet point immediately before the one you're questioning states that gamma penalizes large coefficients, which is not the case (alpha and lambda penalize coefficients, gamma just penalizes the number of leaves). And further down, alpha is the L1 penalty on weights, but the weights are those at each leaf, not on individual features, so alpha does not perform feature selection in the same way as Lasso. (I suppose, though I haven't seen it discussed, that it could force a split candidate's leaf coefficient to zero, causing the algorithm to pass over splitting that feature, perhaps in the long run skipping the feature altogether?)

$\endgroup$
  • $\begingroup$ Please see my post where I tried to explain where the author is coming from. You are right that probably a sentence or two more would be needed to clarify what is going on, but the author's statement in question is not inaccurate. Indeed $\gamma$ is "more relevant" in "shallow tree" use cases. $\endgroup$ – usεr11852 says Reinstate Monic Jul 31 at 19:02
  • 1
    $\begingroup$ @usεr11852, I agree that the author is right in the practical advice: increasing gamma increases regularization, combatting overfitting, and so "gamma brings improvement with shallow trees" could maybe be read as "gamma creates shallower trees"; and the general ideas on the other bullet points work out for practice use as well. But it doesn't seem they have a clear grasp on the parameters. I'd be interested to hear whether you think alpha can (or is likely to) prune out features. $\endgroup$ – Ben Reiniger Jul 31 at 21:08
  • $\begingroup$ I find $\alpha$ being relevant if we use the linear booster. It apparently is relevant for the tree learner too but I have not studied it. As far as I can see, alpha will directly affect the loss as large weights will be penalised. In that sense this probably will result in shallower trees as it will encourage sparsity in the leafs. I somehow suspect it acts a bit as an inverted min_child_weight. (Damn you, I had to spend 30' on this comment. Thanks though, I learned something. Probably this will be a good question on its own right.) $\endgroup$ – usεr11852 says Reinstate Monic Jul 31 at 22:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.