1
$\begingroup$

I am running a convolutional neural network on image data, and returning the gradients in each step yields gradients of exactly zero. At the same time, the network is not converging, and returns high loss.

What does this mean in terms of what I should do to learning rate, momentum, decay, etc.?

Thanks!

Improve this question
$\endgroup$
5
  • 6
    $\begingroup$ What does it mean when the gradient of any function is 0? $\endgroup$
    – Sycorax ♦
    Commented Dec 4, 2016 at 23:24
  • $\begingroup$ I'm using RELU which has f'(x)=0 for x<0 and f'(x)=1 for x>0. $\endgroup$
    – user135237
    Commented Dec 4, 2016 at 23:44
  • 3
    $\begingroup$ I'm familiar with the RELU function. It's possible that all of the neurons have died. However, my question was more general. $\endgroup$
    – Sycorax ♦
    Commented Dec 5, 2016 at 0:06
  • $\begingroup$ I'm not sure of the more general implications -- I just know that it means f'(x)=0, and that parameter would no longer update.... Is there something else you're alluding to? $\endgroup$
    – user135237
    Commented Dec 5, 2016 at 2:17
  • 1
    $\begingroup$ Have a look at stats.stackexchange.com/questions/352036/… $\endgroup$
    – kjetil b halvorsen ♦
    Commented Oct 19, 2021 at 13:01

3 Answers 3

Reset to default
3
$\begingroup$

I am assuming there are no bugs in your code.

Sounds like it might be the Vanishing Gradient Problem.

Try decreasing the depth of your network. Regularization might also help

Improve this answer
$\endgroup$
2
  • 1
    $\begingroup$ or that the model has just found a (local) minima of the cost function. $\endgroup$
    – Dikran Marsupial
    Commented Apr 23, 2022 at 9:26
  • $\begingroup$ Or a local maximum. $\endgroup$
    – Sycorax ♦
    Commented Apr 23, 2022 at 14:50
0
$\begingroup$

Gradients all equal to zero does not necessarily imply any problem with the network. Both minima and maxima occur where the gradient is zero. So it’s possible that your network has arrived at a local minimum or maximum. Determining which is the case requires additional information.

A corner case that is somewhat unlikely is that some combination of RELU units has “died,” so that they give 0s for every input in your data set. But this is somewhat unlikely.

On the other hand, if your network has any bugs or mistakes in the code, then it’s impossible to interpret the model’s results. Always check for bugs.

Improve this answer
$\endgroup$
-2
$\begingroup$

If you're using "tensorflow" and giving your model a batch make sure you put model(batch, training=True). If training isn't set to true then the gradient will be all 0s.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.