1
$\begingroup$

I think I understood the principles of gradient descent and backpropagation. But I think, so far, I'm not sure how they work together.

Gradient descent itself is "just" an optimization method and BP is literally a name for the application of a lot of chain rules. What I wonder is the following:

  1. The output resp. the prediction is measured with a loss function. Then what? How is the error used in the gradient descent approach?
  2. The output consists of lots of inputs from weights and so on. So I guess the final loss function takes into account all these parameters. Will gradient descent then optimize this function (with all parameters)? So it would be a very high-dimensional function to minimize?
Improve this question
$\endgroup$

1 Answer 1

Reset to default
3
$\begingroup$

The output resp. the prediction is measured with a loss function. Then what? How is the error used in the gradient descent approach?

If the output of your neural network is $\hat{y}_\theta$, where $\theta$ refers to the parameters of the net, and if your loss function is $\mathcal{L}(y,\hat{y}_\theta)$, where $y$ is the target, then the gradient descent update rule is $$ \theta_{k+1} \leftarrow \theta_k - \alpha \cdot \nabla_\theta \mathcal{L}(y,\hat{y}_\theta), $$ where $\alpha$ is the learning rate. The "error" and "loss" are the same in this context. You use the backpropagation algorithm to compute $\nabla_\theta \mathcal{L}(y,\hat{y}_\theta)$ exactly, since $\hat{y}_\theta$ is a complicated function of $\theta$.

So I guess the final loss function takes into account all these parameters.

Yes these parameters are in $\theta$.

Will gradient descent then optimize this function (with all parameters)?

Yes. You can think of the gradient $\nabla_\theta\mathcal{L}(y,\hat{y}_\theta)$ as $$ \nabla_\theta\mathcal{L}(y,\hat{y}_\theta) = \frac{\partial \mathcal{L}(y,\hat{y}_\theta)}{\partial \theta}. $$ Then, the gradient can be computed using matrix calculus. Note that if the parameters are stored in matrices and vectors, the gradient can be computed individually for each of these, but the update has to happen to all parameters simultaneously.

So it would be a very high-dimensional function to minimize?

Yes, very.

Improve this answer
$\endgroup$
7
  • $\begingroup$ Tank you very much! This helps a lot! One more question: When I want to change/update a single weight, I would use the above described loss function and derive it by this single weight, right? Or do I use a sum of all partial derivatives which consist of all single weights? $\endgroup$
    – Ben
    Jun 5, 2021 at 13:19
  • 1
    $\begingroup$ To update a single weight, you would apply the gradient descent update rule to that single weight. That is, you would subtract the partial derivative of the loss function with respect to that weight, scaled by $\alpha$, from the weight's current value. $\endgroup$
    – Mahmoud
    Jun 5, 2021 at 13:59
  • $\begingroup$ Thanks a lot again! In practice the gradient descent algorithm is then probably run as many times as there are paremters? And then the loss function is calculated again or is the gradient descent method applied once to a single weight and then the loss function is calculated again? $\endgroup$
    – Ben
    Jun 5, 2021 at 16:55
  • 1
    $\begingroup$ The steps are: 1. Compute partial derivatives of loss function with respect to each parameter individually. If there are $N$ parameters, there will be $N$ partial derivatives. 2. Update the value of each parameter using these $N$ partial derivatives and the gradient descent update rule. There will be a total of $N$ updates. 3. Compute new loss value for a new input. 4. Repeat. $\endgroup$
    – Mahmoud
    Jun 5, 2021 at 16:57
  • 1
    $\begingroup$ Suppose you want to minimize your loss function $f$. You can do this by computing it's gradient, setting it to 0, and then solving for the vector that minimizes it. However, this is most likely not possible, since the solving for this optimal vector is hard. Instead, you can use gradient descent to minimize it, since you do not need to solve for this vector. $\endgroup$
    – Mahmoud
    Jun 14, 2021 at 11:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.