Summation in a Network using identity activation function

Question

I am currently experimenting with a network with one input-layer, one hidden layer, and one output-layer. I am using the identity-function as the activation-function. During the forward-pass, i began computing the outputs of the $j$th neuron in a layer as follows: $$ f \left( \frac{ \sum_{i=1}^n{p_iw_{ji}}}{n} \right) $$ Since $f$ is the identity-function this reduces to: $$ \frac{ \sum_{i=1}^n{p_iw_{ji}}}{n} $$ where $p_i$ is the $i$th input from the previous layer, and $w_{ji}$ is the weight from the $i$th input to the $j$th neuron in the current layer. $n$ is the number of inputs.

But i notice that the more neurons i add, the smaller my output-values become (since the weights and initial values are in the range $[0, 1]$, the resulting values become smaller with each multiplication.

So i tried taking only the sum, without dividing over the number of inputs: $$ \sum_{i=1}^n{p_iw_{ji}} $$ But, unsurprisingly, the resulting values explode with a growing number of neurons.

I know that there are activation-functions that deal with this problem, and keep the values within a certain range, but my question is, whether there are practical ways of keeping the values in a neural net from either becoming very large, or very small, when using a linear activation function.

Ideally, the solution would be one that would not do harm to the network should the activation-function later be swapped.

Answer 1

This problem can be avoided by initializing weights adequately. In general, when using activation-functions that don't clamp the values into a range, the initial weights should be scaled inversely proportional to the number of weights.

For example, when using random values for initial weights,having obtained a random value $r$, a weight could be initialized as $w = \frac{r}{\sqrt{n}}$ where $n$ is the number of weights in the given layer. This way, resulting outputs will likely neither explode nor vanish.

Furthermore, a lower learning-rate may avoid weights being adjusted too much, which otherwise might lead to higher errors in subsequent iterations, and thus, even higher subsequent errors, producing an effect similar to a feedback-loop.