10
$\begingroup$

I'm beginning with deep learning, and I have a question whose answer I couldn't find, maybe I haven't searched properly. I have seen this answer, but it is still not clear what is the weight decay loss and how is it related to the loss function.

$\endgroup$

migrated from ai.stackexchange.com Apr 11 '17 at 19:56

This question came from our site for people interested in conceptual questions about life and challenges in a world where "cognitive" functions can be mimicked in purely digital environment.

9
$\begingroup$

Weight decay specifies regularization in the neural network.

During training, a regularization term is added to the network's loss to compute the backpropagation gradient. The weight decay value determines how dominant this regularization term will be in the gradient computation.

As a rule of thumb, the more training examples you have, the weaker this term should be. The more parameters you have the higher this term should be.

So, Weight decay is a regularization term that penalizes big weights. When the weight decay coefficient is big, the penalty for big weights is also big, when it is small weights can freely grow.

So, now if you go back to reading the answer which you linked in your question, it would make complete sense now.

$\endgroup$
  • 1
    $\begingroup$ I think it is starting to make sense... Thank you! $\endgroup$ – Pablo Arnau González Apr 12 '17 at 10:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.