0
$\begingroup$

I'm trying to implement a convolutional neural network, although for the purposes of this question it could just as well be a fully connected neural network.

Given that each neuron is the sum of the neurons in the previous layer multiplied by the weights, and given that each neuron either propagates this value forward if the value is greater than 0, or propagates 0 otherwise it seems that the activation of neurons through a network should increase. What I'm asking is it it normal for the activations in layer 4 (eg.) to be significantly higher (order of magnitude) than the activations in layer 2.

At the very least, that's what I'm experiencing. I've normalised my inputs, and centered them to mean = 0 but currently I'm still having the issue of exploding activations.

$\endgroup$
1
$\begingroup$

There is no constraint that forces the weights to be close to 1. They can shrink to be nearer zero, which would mean the activations could be near 1, or less.

If your weights are diverging, you might try reducing the learning rate, or trying an optimizer that handles the learning rate for you.

Edit: following your clarification/question below, basically you are probably looked for Xavier initialization, which divides each weight by the sqrt of the fanin, and will mitigate your problem. eg see http://andyljones.tumblr.com/post/110998971763/an-explanation-of-xavier-initialization

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ thanks for your input. I'm asking about the first iteration through a network (ie, before any backprop/weight update scheme has been performed). The weights are randomly selected between -1 and 1. $\endgroup$ – FraserOfSmeg Apr 15 '17 at 21:55
  • $\begingroup$ updated my answer to suggest that Xavier initialization directly addresses your issue. $\endgroup$ – Hugh Perkins Apr 16 '17 at 8:54
0
$\begingroup$

I think there might be a problem with the initialization of the weights. How do you initialize the weights at each layer?

The effect that you describe will only happen if the weights connecting the input units to each output unit are greater than one. However, if your weights are small (initialized accordingly to the number of input neurons) you shouldn't have this problem.

You could try for example Glorot initialization: http://andyljones.tumblr.com/post/110998971763/an-explanation-of-xavier-initialization

| cite | improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.