1
$\begingroup$

I have written an ANN algorithm. And after several iterations my weights grow largely and there's this error which says the value of them is overflowing and therefore the outputs are NaNs. Does it make sense to scale weights for each layer to avoid them from growing very large by dividing them by the largest weight?!

$\endgroup$
1
$\begingroup$

One approach to mitigate this issue could be normalise your inputs. For example, you could subtract the mean of your dataset from each element of your dataset such that the mean of your dataset is zero.

| cite | improve this answer | |
$\endgroup$
  • 1
    $\begingroup$ I have already scaled the inputs by dividing them to the largest, but during training in iterations weights grow very large, I think this happens when I update weights because I have included the momentum expression, and each time it get multiplied by iteration count ... This is my momentum expression: alpha*delta_w*(n-1), alpha is momentum which I considered to be 0.01 and n is number of current iteration, I think when iteration goes high this expression grow... Is it wrong? $\endgroup$ – user5808583 Nov 5 '17 at 0:15
  • 2
    $\begingroup$ Yes, that's the cause of your exploding weights. Your momentum expression should be $\alpha\delta_{n-1}$. That is you store your change in weights from the previous iteration and add that multiplied by your momentum rate (which is 0.01, as you mentioned). $\endgroup$ – ishaanv Nov 5 '17 at 1:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.