1
$\begingroup$

I'm currently reading about neural networks, specifically how loss functions saturation can cause problems. During my studies, I was curious if one could remedy the problem during training of neural networks by detecting saturation (such as comparing the gradients to the previous training iteration) and just increase the learning rate in a proportional manner so that learning still can be done effectively. Unfortunately, my (admittately limited) research on google has not come up with any method using this approach.

Is there a reason why this is not being done?

$\endgroup$
1
$\begingroup$

Saturation means it's not updating . Which means it is on some kind of a local minimum rather than in the desired global minima.

So when increasing the learning rate it can jump our from that local minima and move further .

Refer to the image below about local minima and global minima problems .

enter image description here

Here's a good thread - Improving loss with , increasing learning rate

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.