1
$\begingroup$

There are many questions on this site which have to do with "what happens when two neurons have the same weights/biases" and I am not asking about that.

However, it is occasionally the case that a simple network will converge to a state where two of the neurons have the same weights and biases up to scalar multiplication, representing the same hyperplane dividing the outputs of the previous layer. Are there current methods which avoid this scenario, and is it desirable to avoid this scenario?

I don't believe regularization would prevent this, since this "degenerate weight state" would not lead to overfitting.

$\endgroup$
1
$\begingroup$

This might occur due to poor initialization of network weights or some other reason; and it's, in general, desirable to prevent it since you're wasting network resources. Dropout directly addresses this problem (along with and many others), since it randomly shuts down some of the neurons, breaking up the symmetry. A related, but not the same problem in literature is co-adaptation, where dropout has been shown to be successful. The [complex] co-adaptation is a harder one to tackle with, where a group of neurons learn such that they might cancel each others' effects, or be highly dependent on each others' outputs. Learning same weights/biases could be just a subset of this wider issue; where dropout should be useful addressing it in my opinion.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.