I'm beginning with deep learning, and I have a question whose answer I couldn't find, maybe I haven't searched properly. I have seen this answer, but it is still not clear what is the weight decay loss and how is it related to the loss function.
migrated from ai.stackexchange.com Apr 11 '17 at 19:56
This question came from our site for people interested in conceptual questions about life and challenges in a world where "cognitive" functions can be mimicked in purely digital environment.
Weight decay specifies regularization in the neural network.
During training, a regularization term is added to the network's loss to compute the backpropagation gradient. The
weight decay value determines how dominant this regularization term will be in the gradient computation.
As a rule of thumb, the more training examples you have, the weaker this term should be. The more parameters you have the higher this term should be.
So, Weight decay is a regularization term that penalizes big weights. When the weight decay coefficient is big, the penalty for big weights is also big, when it is small weights can freely grow.
So, now if you go back to reading the answer which you linked in your question, it would make complete sense now.