When using RELU is it normal for the activations to go up at each layer I'm trying to implement a convolutional neural network, although for the purposes of this question it could just as well be a fully connected neural network.
Given that each neuron is the sum of the neurons in the previous layer multiplied by the weights, and given that each neuron either propagates this value forward if the value is greater than 0, or propagates 0 otherwise it seems that the activation of neurons through a network should increase. What I'm asking is it it normal for the activations in layer 4 (eg.) to be significantly higher (order of magnitude) than the activations in layer 2.
At the very least, that's what I'm experiencing. I've normalised my inputs, and centered them to mean = 0 but currently I'm still having the issue of exploding activations.
 A: There is no constraint that forces the weights to be close to 1. They can shrink to be nearer zero, which would mean the activations could be near 1, or less.
If your weights are diverging, you might try reducing the learning rate, or trying an optimizer that handles the learning rate for you.
Edit: following your clarification/question below, basically you are probably looked for Xavier initialization, which divides each weight by the sqrt of the fanin, and will mitigate your problem. eg see http://andyljones.tumblr.com/post/110998971763/an-explanation-of-xavier-initialization
A: I think there might be a problem with the initialization of the weights. How do you initialize the weights at each layer?
The effect that you describe will only happen if the weights connecting the input units to each output unit are greater than one. However, if your weights are small (initialized accordingly to the number of input neurons) you shouldn't have this problem.
You could try for example Glorot initialization: http://andyljones.tumblr.com/post/110998971763/an-explanation-of-xavier-initialization
