7
$\begingroup$

I have a decent grasp of neural networks, back propagation and chain rule however I am struggling to understand auto differentiation.

The below refer to auto differentiation outside the context of back propagation:

  • How does auto differentiation compute the gradient from a matrix?
  • What are the requirements to compute a gradient? Does a function need to be specified?
  • What are some use cases for this (other then back propagation)?
  • Why is it important and what are the alternatives?

Am I missing something?

Improve this question
$\endgroup$
4
  • 3
    $\begingroup$ Automatic differentiation, also known as algorithmic differentiation, is an automated way of numerically calculating derivatives of a function(s) specified by a computer program, but the functions can be indirectly defined by the computer program. It is useful for computing gradients, Jacobians, and Hessians for use in numerical optimization, among other things. Backpropagation is an implementation of the reverse mode of automatic differentiation for computing the gradient of a neural network optimization problem See en.wikipedia.org/wiki/Automatic_differentiation . $\endgroup$
    – Mark L. Stone
    Aug 21, 2016 at 15:03
  • 3
    $\begingroup$ Alternatives are symbolic differentiation and finite differences, both of which are usually slower, and finite differences might be less accurate. Hand-coded derivatives can be used if the human can figure out the derivative, but are prone to human mistakes in many cases. $\endgroup$
    – Mark L. Stone
    Aug 21, 2016 at 15:05
  • $\begingroup$ Where can I find simple examples to see how it is implemented? Is it the tf.gradient method I should be looking at? $\endgroup$
    – Greg
    Aug 21, 2016 at 15:08
  • 3
    $\begingroup$ See list of automatic differentiation software and tools at Wikipedia link, and also links which include example usage. Another alternative to automatic differentiation is complex step differentiation (derivative) aero-comlab.stanford.edu/Papers/martins.aiaa.01-0921.pdf , which winds up being almost the same thing as the forward mode of automatic differentiation. $\endgroup$
    – Mark L. Stone
    Aug 21, 2016 at 15:13

1 Answer 1

Reset to default
12
$\begingroup$

In auto differentiation systems mostly an operator (e.g. addition, subtraction) is defined together with its differentiation.

So after you write a function by stacking a series of operators, the system can figure out by itself how the corresponding derivatives should be computed, usually by using computation graphs and the chain rule.

Auto differentiation is beneficial for gradient based optimization (e.g. training a neural network using gradient descent), as it saves us from working out the math, implementing the code and verifying the derivatives numerically case by case.

Here's how to define an operator (op) in Theano and Tensorflow.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.