How does regularization part of w help in XGBoost

Question

In the regularization part of XGBoost objective function, it contains gammaT and also lambdasquare(W).

I understand gamma is the minimum node split criteria and T is number of leaves and regularizing them lead to a simpler model (not many splits/leaves).

However, I don't understand why regularizing /penalizing w will help in simpler model or what it does ? Since we are fitting regression trees for both regression and classification in XGBoost, how does having a small score on the leaf help with regularization of the tree structure? Or w is just there not regularize the tree structure but to keep the leaf weights low (if so, why)?

Thank you for any responses.

Answer 1

The regularization primarily reduces the magnitude of the leaf weights, yes. When using L1 regularization, that can make a potential split no longer worth it, and the tree may end up being smaller than otherwise (see this colab notebook, originally drafted for my answer to this other question). L2 regularization can do the same, but not nearly as aggressively (again see the notebook).

Much more directly, regularization shrinks the scores, making the final aggregate prediction more conservative, helping to combat overfitting.