1
$\begingroup$

I am having trouble understanding the concepts of weight decay and weight sharing for ANN's. Would someone mind explaining these two things?

$\endgroup$
2
$\begingroup$

Weight decay is an alteration to backpropagation, seeking to avoid overfitting, in which weights are decreased by a small factor during each iteration. From Mitchell Machine Learning, p. 111:

This is equivalent to modifying the definition of $E$ [error function] to include a penalty term corresponding to the total magnitude of the network weights. The motivation for this approach is to keep weight values small, to bias learning against complex decision surfaces.

Weight sharing is when different units in the network literally share the same weights: They use identical values, "usually to enforce some constraint known in advance to the human designer." (Ibid. p. 118)

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.