6
$\begingroup$

I've understood how the backpropagation algorithm uses the partial derivatives of the weights to train a normal neural network. However, I cannot quite understand how the algorithm changes the filters. Is it in the same way i.e. does the backpropagation algorithm find the derivates of the filters?

$\endgroup$
6
$\begingroup$

First of all: 1) The filters are the weights, 2) As justin pointed out, backpropagation is simply a method for computing gradients. You update the weights using another optimization algorithm such as gradient descent. Gradient descent updates the weights at each time step. It updates the weights using the negative gradient of the loss function. The partial derivatives that are being calculated are not those of the weights, but of the loss function with respect to the weights:

$$ w_{t+1} = w_{t} - \epsilon \frac{\partial \mathcal{L}(w)}{\partial w} \\ $$

$\endgroup$
  • 1
    $\begingroup$ Nit pick: even though back propagation has been used with gradient descent all the time, it is actually an efficient way to calculate the gradients. Whether gradient descent is performed afterwards or conjugate gradient or rmsprop or whatever is not important. $\endgroup$ – bayerj Mar 11 '15 at 8:43
  • $\begingroup$ You're right. It depends on whether he was using backpropagation to mean only the calculation of the gradient for later use in gradient descent, or another optimization algorithm as you've suggested. Or, if he was using it in a more casual and general sense as a learning algorithm. Either way, I've updated the post to clarify. $\endgroup$ – sabalaba Mar 11 '15 at 9:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.