8
$\begingroup$

I use LightGBM for regression task and I'm planning to use L2-regularization to avoid overfitting. It's quite clear for me what L2-regularization does in linear regression but I couldn't find any information about its use in LightGBM. What exactly does L2-regularization in LightGBM or other Gradient Boosting algorithms do?

$\endgroup$
1
$\begingroup$

It does basicly the same. It penalizes the weights upon training depending on your choice of the LightGBM L2-regularization parameter 'lambda_l2', aiming to avoid any of the weights booming up to a level that can cause overfitting, suppressing the variance of the model. Regularization term again is simply the sum of the Frobenius norm of weights over all samples multiplied by the regularization parameter lambda and divided by the number of samples. You add this to the cost function of the machine learning algorithm that you work on just like linear regression. If you want to have a mathematical landscape specificly for LightGBM, you can refer to the LightGBM paper published at Conference on Neural Information Processing Systems (NIPS) 2017 to see the cost function which Microsoft Team have designed for the boosting algorithm:

https://papers.nips.cc/paper/6907-lightgbm-a-highly-efficient-gradient-boosting-decision-tree.pdf

Just imagine or express manually that you add the regularization term to the cost function just like it is done on the cost function of the linear regression.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.