I am training a deep neural network with several convolutional layers and a fully connected layer at the bottom and I am generating histograms of the weight distributions to try and understand how the network is training.
When looking at the graphs, I found something puzzling: most of the weights are near zero and only a small portion of the weights are getting very large. Why is this happening? Is this good and expected, or is this undesirable? Although I have only posted two example layers, this is happening throughout my network.
- Data is very sparse and nearly binary (mostly 1's, very few 0's)
- Input is normalized to be in the range 0-1
- Not using L1/L2 yet since weights are mostly small
- Activations are all leaky Relu (a=0.3)
- I am performing batch normalization after each preactivation