# Neural network with decay, other forumula's possible?

I am following the coding tutorials on MSDN to experiment with neural networks.

After observing and testing various test datasets and neural net behaviours, I got curious. Some networks use decay functions, the way I see that is to more rapidly advance the early part of training epochs. The decay usually is based upon some factor on the epoch number.

I see how neural networks, adapt and decrease their error rate as an approaching problem. And that made me wonder whether there are functions (or neural nets) that use a different forms of decay.

For example if its error is 50% (0.5) it uses 0.001 as decay value, and if the error gets 95% (0.01052631) it would use 0.00001. So always two (or three) decimals lower then error rate. I'm just wondering this. Because sometimes in the end of neural network training doesn't progress that much anymore and can start to oscillate, then I'm often thinking well why doesn't it just try with a smaller changes?

Is there such a function? And is this used among weight decay networks or not ? Cause i'm wondering if my observation and understanding of this is right or wrong.

## 1 Answer

You can have a time-varying regularization parameter (lambda) tied to the epoch or to the error rate. For example, in TensorFlow, the weight regularization can be implemented as follows:

loss = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(logits, train_labels))
regularizers = tf.nn.l2_loss(weights_1) + tf.nn.l2_loss(weights_2)
loss = tf.reduce_mean(loss + lambda * regularizers)


Here we are assuming the same lambda is used for each layer. The lambda can be made time-varying according to a performance criteria in order to make regularization time-varying.