I am currently experimenting with a network with one input-layer, one hidden layer, and one output-layer. I am using the identity-function as the activation-function. During the forward-pass, i began computing the outputs of the $j$th neuron in a layer as follows: $$ f \left( \frac{ \sum_{i=1}^n{p_iw_{ji}}}{n} \right) $$ Since $f$ is the identity-function this reduces to: $$ \frac{ \sum_{i=1}^n{p_iw_{ji}}}{n} $$ where $p_i$ is the $i$th input from the previous layer, and $w_{ji}$ is the weight from the $i$th input to the $j$th neuron in the current layer. $n$ is the number of inputs.

But i notice that the more neurons i add, the smaller my output-values become (since the weights and initial values are in the range $[0, 1]$, the resulting values become smaller with each multiplication.

So i tried taking only the sum, without dividing over the number of inputs: $$ \sum_{i=1}^n{p_iw_{ji}} $$ But, unsurprisingly, the resulting values explode with a growing number of neurons.

I know that there are activation-functions that deal with this problem, and keep the values within a certain range, but my question is, whether there are practical ways of keeping the values in a neural net from either becoming very large, or very small, when using a linear activation function.

Ideally, the solution would be one that would not do harm to the network should the activation-function later be swapped.

  • 1
    $\begingroup$ Are you aware that with identity activation adding more layers to network has same effect as widening single-layer network? $\endgroup$
    – Tim
    Commented Jul 11, 2020 at 9:40
  • $\begingroup$ @Thanks for the hint, intuitively this makes sense, i got to do the math on it to see for myself if it really is equivalent. $\endgroup$ Commented Jul 12, 2020 at 21:19

1 Answer 1


This problem can be avoided by initializing weights adequately. In general, when using activation-functions that don't clamp the values into a range, the initial weights should be scaled inversely proportional to the number of weights.

For example, when using random values for initial weights,having obtained a random value $r$, a weight could be initialized as $w = \frac{r}{\sqrt{n}}$ where $n$ is the number of weights in the given layer. This way, resulting outputs will likely neither explode nor vanish.

Furthermore, a lower learning-rate may avoid weights being adjusted too much, which otherwise might lead to higher errors in subsequent iterations, and thus, even higher subsequent errors, producing an effect similar to a feedback-loop.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.