I have different activation functions in my network. I have noticed my networks failing (producing
NaN). The reasoning behind this is:
- I have a large layers with average weights at start, so some neurons get large values as input
- Softplus activation function outputs
- Sinusoid/Softsign/Bent identity produces
NaNas output to
How can I stop this from happening? Shoud I put limits on inputs (e.g.
Also, the derivative of the complementary log-log activation function also returns
Infinity rather quickly (even though the output of the function is limited), causing the backpropagation algorithm to fail. I solved this by returning 0 if x > 800.
And last of all, I have nodes with the
Absolute activation function. When these nodes are selfconnected, their activations will infinitely keep getting larger -> after a while they will output
Infinity as well.