2
$\begingroup$

LeNet has several fully connected layers, I'm wondering what prevents neurons duplicate other's weights and outputs.

Unfortunately the only technique, I came up with, is random weight initialization (luck). And may be dropout... but it's random as well.

Is there way to detect such situations and remove duplicates.

$\endgroup$
1
  • $\begingroup$ regularization may do it, but I don't know more about it. L1 regularization should prevent having multiple correlated features, but I don't know what will happen with neurons $\endgroup$
    – rep_ho
    Jan 26 '16 at 14:45
1
$\begingroup$

Typically random weight initialization is effective at breaking symmetry in neural nets and we do learn different neurons at each layer. Now, different might only be true in the sense of numerical precision.

What I have definitely seem happen is a related issue, and that's of highly correlated features, so maybe the weights aren't exactly the same in a numerical precision sense, but they either qualitatively or quantitatively via some correlation measure are very similar and are redundant. This may represent a waste of representational capacity.

To this end, a recent ICLR 2016 submission delves into this issue for Conv Nets: Reducing Overfitting in Deep Networks by Decorrelating Representations. This may be of interest to you as they use a regularization term to induce decorrelation in their conv net features.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.