# Summation in a Network using identity activation function

I am currently experimenting with a network with one input-layer, one hidden layer, and one output-layer. I am using the identity-function as the activation-function. During the forward-pass, i began computing the outputs of the $$j$$th neuron in a layer as follows: $$f \left( \frac{ \sum_{i=1}^n{p_iw_{ji}}}{n} \right)$$ Since $$f$$ is the identity-function this reduces to: $$\frac{ \sum_{i=1}^n{p_iw_{ji}}}{n}$$ where $$p_i$$ is the $$i$$th input from the previous layer, and $$w_{ji}$$ is the weight from the $$i$$th input to the $$j$$th neuron in the current layer. $$n$$ is the number of inputs.

But i notice that the more neurons i add, the smaller my output-values become (since the weights and initial values are in the range $$[0, 1]$$, the resulting values become smaller with each multiplication.

So i tried taking only the sum, without dividing over the number of inputs: $$\sum_{i=1}^n{p_iw_{ji}}$$ But, unsurprisingly, the resulting values explode with a growing number of neurons.

I know that there are activation-functions that deal with this problem, and keep the values within a certain range, but my question is, whether there are practical ways of keeping the values in a neural net from either becoming very large, or very small, when using a linear activation function.

Ideally, the solution would be one that would not do harm to the network should the activation-function later be swapped.

• Are you aware that with identity activation adding more layers to network has same effect as widening single-layer network?
– Tim
Jul 11, 2020 at 9:40
• @Thanks for the hint, intuitively this makes sense, i got to do the math on it to see for myself if it really is equivalent. Jul 12, 2020 at 21:19

For example, when using random values for initial weights,having obtained a random value $$r$$, a weight could be initialized as $$w = \frac{r}{\sqrt{n}}$$ where $$n$$ is the number of weights in the given layer. This way, resulting outputs will likely neither explode nor vanish.