9
$\begingroup$

I am training a deep neural network with several convolutional layers and a fully connected layer at the bottom and I am generating histograms of the weight distributions to try and understand how the network is training.

When looking at the graphs, I found something puzzling: most of the weights are near zero and only a small portion of the weights are getting very large. Why is this happening? Is this good and expected, or is this undesirable? Although I have only posted two example layers, this is happening throughout my network.

Weight distribution histograms

Additional information:

  • Data is very sparse and nearly binary (mostly 1's, very few 0's)
  • Input is normalized to be in the range 0-1
  • Not using L1/L2 yet since weights are mostly small
  • Activations are all leaky Relu (a=0.3)
  • I am performing batch normalization after each preactivation
$\endgroup$
2
  • 2
    $\begingroup$ What type of data are you using? Is it normalized to a range of [0,1] or did you make your data have a mean of 0 with a std of 1? Are you using any type of weight regularization (L1/L2)? What activation functions are you using? $\endgroup$ Feb 26, 2016 at 21:25
  • 1
    $\begingroup$ It would be interesting to see whether if you pruned (set to 0) all small weights, the network would keep the same accuracy. $\endgroup$
    – hans
    Oct 23, 2020 at 16:37

2 Answers 2

7
$\begingroup$

This is expected. Weights in a CNN form feature detectors, so that a certain pattern in an image is connected to strong weights, but the rest of the image pixels should not cause any activations in the next layer neurons.

Only a small fraction of neurons in a layer is activated every time an image is shown, and a small fraction of weights is needed to be large to activate (or suppress) any particular neuron. Moreover, the number of patterns a network needs to detect is fairly small, especially in the early layers. Therefore, overall connectivity for the network is usually very sparse.

The same reasoning applies to regulatization methods, such as L2/L1 - forcing the weights to be small makes the network more robust to noise in the data, and forces the network to learn only the features present in many images.

$\endgroup$
0
$\begingroup$

This is a really good question and there can be multiple explanations to this phenomenon. I will try to give out a non-exhaustive list for the same.

  1. Sparse data ? You mean sparse/binary feature set ? hmmm...this can happen. See in that case your network does not require much parameters to encode this function. Every neural network is a function that encodes the feature set to the set of labels (one hot or whatever !).

Now lets talk about regularization because that has soooo much to do about the weight distribution of a network.

  1. You are not using dropout. Try using it. It is a great regularization technique which can actually help you to use all of your neurons. Without dropout on Sparse data, this is an expected event.

  2. Try using L2, it might have some effect to your weight distribution.

You are using batchnorm and still getting this ? Hmm...I dont have an answer right now but if this post generates enough attention I will do a study on dropout vs batchnorm...!!

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.