1
$\begingroup$

I have a question about the gradient descent step in neural networks. I fully understand the derivative step and taking the steps required to move in the direction that reduces the loss (finding the minima).

What i am unsure about is the direction in which we are moving. When calculating the derivative of the the cost with respect to the weights, why is it that we move in the direction of minimizing this cost? why does this optimize the model and cause weight adjustment that produces better predictions.

Is the local minima directly linked to the loss function?

Is it intuitive that the loss function creates error metric 'x' and that finding the local maximum through gradient descent for example would simply increase the error and thus weight adjustment moves in the wrong direction?

Improve this question
$\endgroup$
5
  • $\begingroup$ The cost measures how good/bad the model is, so lowering the cost implies that the model is better (at least wrt the data you used to measure cost). Or are you asking something else? (What?) $\endgroup$
    – Sycorax ♦
    Dec 2, 2019 at 17:19
  • $\begingroup$ thank you! yes sort of! is this the intuition for why therefore, we move in the direction of the minima when calculating the derivative of the loss with respect to the weights? as this would proportionally reduce the error and therefore optimise the model further? $\endgroup$
    – user9317212
    Dec 2, 2019 at 17:25
  • $\begingroup$ I think a good start is en.wikipedia.org/wiki/Gradient_descent $\endgroup$
    – crlb
    Dec 2, 2019 at 17:26
  • 1
    $\begingroup$ This is simply tautological: if your cost is lower, the model has improved because the cost is lower. If the cost is higher, the model is worse. You question seems to be asking "If I pick up \$20 on the street, am I richer?" Well, yes, obviously you're richer because you have an additional \$20. But one quirk is that the gradient doesn't always (or even usually) point at a minimum -- even though you can lower the cost anyway. stats.stackexchange.com/questions/367397/… $\endgroup$
    – Sycorax ♦
    Dec 2, 2019 at 17:30
  • $\begingroup$ Thank you! This indeed helps clear it up! $\endgroup$
    – user9317212
    Dec 2, 2019 at 17:40

1 Answer 1

Reset to default
2
$\begingroup$

Before I begin my answer and because you use both terms in your question, I'd like to note that:

$$ loss \; function = cost \ function $$

The cost function is the way your model evaluates its performance. The lower the cost, the closer your predictions are to the targets and thus the better your model is performing. This is the definition of the loss function. So, the goal of the whole learning process is to simply lower the value of the cost function.

How do you proceed to minimize the cost? SGD! Once you reach a local minima and the cost is at its lowest value, you can say that that's the closest your predictions can get to the targets.

In case something isn't exactly clear I'll answer your questions one by one:

When calculating the derivative of the the cost with respect to the weights, why is it that we move in the direction of minimising this cost?

Because that's the goal of the whole training procedure: to minimize the cost. The cost shows you how close your predictions are to the targets.

Why does this optimise the model and cause weight adjustment that produces better predictions. Is the local minima directly linked to the loss function?

Yes the local minimum is referring to the loss function. I.e. it's a place in the function where, whichever direction you choose will increase its value.

Is it intuitive that the loss function creates error metric 'x' and that finding the local maximum through gradient descent for example would simply increase the error and thus weight adjustment moves in the wrong direction?

No, the (training) error is measured by the loss function. By minimizing the loss function, by definition, you are reducing the error.

Improve this answer
$\endgroup$
2
  • 1
    $\begingroup$ The only caveats I would add is that (1) the loss can increase if the step size is too large and (2) decreasing loss on the training set can increase the loss on some hold-out set (i.e. you can overfit). $\endgroup$
    – Sycorax ♦
    Dec 3, 2019 at 21:41
  • $\begingroup$ Thank you so much for this very clear explanation! $\endgroup$
    – user9317212
    Dec 3, 2019 at 22:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.