I'm beginning with deep learning, and I have a question whose answer I couldn't find, maybe I haven't searched properly. I have seen this answer, but it is still not clear what is the weight decay loss and how is it related to the loss function.
$\begingroup$
$\endgroup$
1
asked Apr 11, 2017 at 16:33
-
$\begingroup$ youtube.com/watch?v=6g0t3Phly2M&ab_channel=DeepLearningAI This also is a good and short video of Andrew Ng that talks about weight decay $\endgroup$– Mas ACommented Dec 21, 2021 at 17:14
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
1
Weight decay specifies regularization in the neural network.
During training, a regularization term is added to the network's loss to compute the backpropagation gradient. The weight decay value determines how dominant this regularization term will be in the gradient computation.
As a rule of thumb, the more training examples you have, the weaker this term should be. The more parameters you have the higher this term should be.
So, Weight decay is a regularization term that penalizes big weights. When the weight decay coefficient is big, the penalty for big weights is also big, when it is small weights can freely grow.
So, now if you go back to reading the answer which you linked in your question, it would make complete sense now.
-
1$\begingroup$ I think it is starting to make sense... Thank you! $\endgroup$ Commented Apr 12, 2017 at 10:00