I'm trying to implement a convolutional neural network, although for the purposes of this question it could just as well be a fully connected neural network.
Given that each neuron is the sum of the neurons in the previous layer multiplied by the weights, and given that each neuron either propagates this value forward if the value is greater than 0, or propagates 0 otherwise it seems that the activation of neurons through a network should increase. What I'm asking is it it normal for the activations in layer 4 (eg.) to be significantly higher (order of magnitude) than the activations in layer 2.
At the very least, that's what I'm experiencing. I've normalised my inputs, and centered them to mean = 0 but currently I'm still having the issue of exploding activations.