0
$\begingroup$

In neural net training, nowadays tanh and sigmoid activation functions in hidden layers are avoided as they tend to "saturate" easily. Meaning, if the x value plugged into tanh/sigmoid is very large or very small, The derivative at that value will tend to zero and thus the changes we derive from that gradient to the neural network will also be small.

tanh and sigmoid functions and their derivitives respectively

However -- We apply changes to our NN from this gradient by multiplying it by a learning rate. If an "ideal" gradient derivative function is the weight itself (ie. a derivative of a linear function), Is there a reason people don't artificially inflate a learning rate for more saturated derivative values so those neurons are able to change easily while retaining the direction of the gradient?

Improve this question
$\endgroup$
2
  • 1
    $\begingroup$ This is essentially what, for instance, Adam does. See e.g. stats.stackexchange.com/questions/220494/… $\endgroup$
    – John Madden
    Commented Nov 26 at 16:59
  • $\begingroup$ @JohnMadden That makes sense! Which leads me to ask: 1) if Adam's momentum per parameter is tracked via a exponentially weighted average, if a "gradient momentum" would approach zero, does common convention say that an optimizer is fairly "stopping to optimize" a parameter that's mostly optimized? 2) Are you aware of cases where a gradient is "pre-inflated" to assist in even prameter training (such that it effectively becomes a constant), or does the magnitude of a given derivative important? $\endgroup$
    – Omrii
    Commented Nov 27 at 17:37

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.