1
$\begingroup$

hello I have a question about weight regularization in Adam apparently the weight_decay in the AdamW function https://huggingface.co/transformers/main_classes/optimizer_schedules.html#adamw-pytorch has the same impact as an L2 regularization

my questions are: is that parameter the same as lambda that we have in the regularization term?

L2 regularization formula

how does it exactly work? and what is its impact on the model complexity?

$\endgroup$
1
  • $\begingroup$ Everything is explained in arxiv.org/abs/1711.05101. PyTorch used AdamW, as well some other libraries at this moment. FastAi was the first library implementing AdamW. Your question has been answered in here. $\endgroup$
    – prosti
    Jul 21, 2020 at 22:05

1 Answer 1

2
$\begingroup$

apparently the weight_decay in the AdamW function [...] has the same impact as an L2 regularization

This claim isn't entirely correct. Both of these regularization techniques are conceptually, but they aren't the same in the case of adaptive gradient algorithms.

In fact, the AdamW paper begins by stating:

L2 regularization and weight decay regularization are equivalent for standard stochastic gradient descent (when rescaled by the learning rate), but as we demonstrate this is not the case for adaptive gradient algorithms, such as Adam.

For more information about how it works I suggest you read the paper.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.