-1
$\begingroup$

I have trained a recurent neural network on tensirflow so there is no need t initialize my paraemters,it will be done automatically into the tf.dynamic_rnn so when i train my model ,i don't get right predictions all the time.I mean when i compile my code,i get sometimes good predictions but sometimes my model fail so i think this is due to the fact that gradient descent is sometimes stuck on a local minimum so what can i do , i did the tf.train.AdamOPtimizer(0.01).minimize(loss) .what is the best optimizer that avoids local minimum

$\endgroup$

2 Answers 2

1
$\begingroup$

When neural networks are stuck in a local minimum the problem is usually the activation function. Which one works best? That changes from project to project. Most of the time, we find the best activation function by trial and error...welcome to world of machine learning!

I seriously over-complicated this in the beginning. I researched for hours and hours which activation function is "best" or when to use each one. The answer was so simple that I couldn't believe it. The answer I received from many, many modelers without hesitation or variation was "whichever one works best."

Hope this helps.

$\endgroup$
1
  • 1
    $\begingroup$ It's scary but it's true! Just to add to Sarah, it's always good to understand every hyperparameter that can be tweaked. If the problem seems simple enough, there should be a set of parameters that allow your model to converge well nearly all the time. $\endgroup$
    – djma
    Jul 21, 2017 at 20:27
0
$\begingroup$

in fact ,in tensorflow ,i created an RNN cell and it sets automatically the activation function so i don t think about changing my activation function, i replaced the gradient descent optimize by the momentum optimizer and i set a momentum of 0.001 so that way it minimizes the chances that it get stuck in a local minimum

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.