# ANN: Weights grow very large, can I scale them?

I have written an ANN algorithm. And after several iterations my weights grow largely and there's this error which says the value of them is overflowing and therefore the outputs are NaNs. Does it make sense to scale weights for each layer to avoid them from growing very large by dividing them by the largest weight?!

• Yes, that's the cause of your exploding weights. Your momentum expression should be $\alpha\delta_{n-1}$. That is you store your change in weights from the previous iteration and add that multiplied by your momentum rate (which is 0.01, as you mentioned). – ishaanv Nov 5 '17 at 1:54