Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It only takes a minute to sign up.
Sign up to join this community
LeNet has several fully connected layers, I'm wondering what prevents neurons duplicate other's weights and outputs.
Unfortunately the only technique, I came up with, is random weight initialization (luck). And may be dropout... but it's random as well.
Is there way to detect such situations and remove duplicates.
Typically random weight initialization is effective at breaking symmetry in neural nets and we do learn different neurons at each layer. Now, different might only be true in the sense of numerical precision.
What I have definitely seem happen is a related issue, and that's of highly correlated features, so maybe the weights aren't exactly the same in a numerical precision sense, but they either qualitatively or quantitatively via some correlation measure are very similar and are redundant. This may represent a waste of representational capacity.
To this end, a recent ICLR 2016 submission delves into this issue for Conv Nets: Reducing Overfitting in Deep Networks by Decorrelating Representations. This may be of interest to you as they use a regularization term to induce decorrelation in their conv net features.
