I have a decent grasp of neural networks, back propagation and chain rule however I am struggling to understand auto differentiation.

The below refer to auto differentiation outside the context of back propagation:

  • How does auto differentiation compute the gradient from a matrix?
  • What are the requirements to compute a gradient? Does a function need to be specified?
  • What are some use cases for this (other then back propagation)?
  • Why is it important and what are the alternatives?

Am I missing something?

  • 3
    $\begingroup$ Automatic differentiation, also known as algorithmic differentiation, is an automated way of numerically calculating derivatives of a function(s) specified by a computer program, but the functions can be indirectly defined by the computer program. It is useful for computing gradients, Jacobians, and Hessians for use in numerical optimization, among other things. Backpropagation is an implementation of the reverse mode of automatic differentiation for computing the gradient of a neural network optimization problem See en.wikipedia.org/wiki/Automatic_differentiation . $\endgroup$ – Mark L. Stone Aug 21 '16 at 15:03
  • 3
    $\begingroup$ Alternatives are symbolic differentiation and finite differences, both of which are usually slower, and finite differences might be less accurate. Hand-coded derivatives can be used if the human can figure out the derivative, but are prone to human mistakes in many cases. $\endgroup$ – Mark L. Stone Aug 21 '16 at 15:05
  • $\begingroup$ Where can I find simple examples to see how it is implemented? Is it the tf.gradient method I should be looking at? $\endgroup$ – Greg Aug 21 '16 at 15:08
  • 3
    $\begingroup$ See list of automatic differentiation software and tools at Wikipedia link, and also links which include example usage. Another alternative to automatic differentiation is complex step differentiation (derivative) aero-comlab.stanford.edu/Papers/martins.aiaa.01-0921.pdf , which winds up being almost the same thing as the forward mode of automatic differentiation. $\endgroup$ – Mark L. Stone Aug 21 '16 at 15:13

In auto differentiation systems mostly an operator (like addition, subtraction) is defined together with its differentiation.

So after you write a function by stacking a series of operators, the program can figure out by itself how the corresponding derivatives should be computed (often by using some computation graph and the chain rule).

The benefit is obvious as it saves us from working out the math, implementing the code and verifying the derivatives numerically case by case.

Here's how to define an operator (op) in Theano and Tensorflow.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.