1
$\begingroup$

Going through this book, I am familiar with the following:

For each training instance the backpropagation algorithm first makes a prediction (forward pass), measures the error, then goes through each layer in reverse to measure the error contribution from each connection (reverse pass), and finally slightly tweaks the connection weights to reduce the error.

However I am not sure how this differs from the reverse-mode autodiff implementation by TensorFlow. As far as I know the above algorithm first goes through the graph in the forward direction and then in the second pass computes all partial derivatives for the outputs with respect to the inputs. This is very similar to the propagation algorithm.

How does backpropagation differ from reverse-mode autodiff ?

Improve this question
$\endgroup$
1
  • $\begingroup$ I'm not an expert but I believe it's basically the same thing, except backprop also includes the step of actually updating the weights. $\endgroup$
    – shimao
    Apr 19, 2018 at 18:07

1 Answer 1

Reset to default
2
$\begingroup$

Thanks for the above comment. I have found the answer to this question by the author of the book himself:

Bakpropagation refers to the whole process of training an artificial neural network using multiple backpropagation steps, each of which computes gradients and uses them to perform a Gradient Descent step. In contrast, reverse-mode auto diff is simply a technique used to compute gradients efficiently and it happens to be used by backpropagation.

Improve this answer
$\endgroup$
1
  • $\begingroup$ I agree, in the end you end up with many many partial derivatives (in backpropagation). You can calculate them as you wish without actually using reverse-mode autodiff. You can use feedforward-Mode autodiff or any other algorithm you prefer. The reverse mode just happens to be the best for that. It may be also confusing (why you got the question in the first place) because reverse-mode autodiff works in a similar "philosophy" as backpropagation, it computes first all the values (feedforwar phase) and then goes back to compute all the partial derivatives at once. $\endgroup$
    – Agustin Barrachina
    Jul 18, 2019 at 13:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.