I'm following the Tensorflow mnist tutorial (https://github.com/tensorflow/tensorflow/blob/master/tensorflow/examples/tutorials/mnist/mnist_softmax.py).

The tutorial uses tf.train.Optimizer.minimize (specifically tf.train.GradientDescentOptimizer). I don't see any arguments being passed in anywhere to define gradients.

Is Tensor flow using numerical differentiation by default?

Is there a way to pass in gradients like you can with scipy.optimize.minimize?


It's not numerical differentiation, it's automatic differentiation. This is one of the main reasons for tensorflow's existence: by specifying operations in a tensorflow graph (with operations on Tensors and so on), it can automatically follow the chain rule through the graph and, since it knows the derivatives of each individual operation you specify, it can combine them automatically.

If for some reason you want to override that piecewise, it's possible with gradient_override_map.

  • $\begingroup$ Isn't automatic differentiation using numerical differentitation? $\endgroup$ – Aerin Jun 21 '17 at 21:39
  • $\begingroup$ @BYOR No; check out the Wikipedia link above. What tensorflow does is actually somewhere in between "real" reverse-mode autodiff and symbolic differentiation, though. $\endgroup$ – Dougal Jun 21 '17 at 21:42
  • $\begingroup$ @Dougal im using eager mode with no graph, no tf.function decorator, how does the auto differentation know the relation between tensors? $\endgroup$ – datdinhquoc Oct 1 '19 at 10:19
  • 1
    $\begingroup$ @datdinhquoc It's the same fundamental algorithm, just a little more complicated of an implementation. $\endgroup$ – Dougal Oct 1 '19 at 12:22

It uses automatic differentiation. Where it uses chain rule and go backword in the graph assigning gradients.

Let’s say we have a tensor C This tensor C has made after series of operations Let’s say by adding , multiplying , going through some nonlinearity etc

So if this C depends on some set of tensors called Xk , We need to get the gradients

Tensorflow always track the path of operations. I mean the sequential behavior of the nodes and how data flow between them. That is done by the graph enter image description here

If we need to get the derivatices of the cost w.r.t X inputs what this will first does is it load the path from x-input to the cost by extending the graph.

Then it start in the rivers order. Then distribute the gradients with chain rule. (Same as backpropagation)

Any way if you read the source codes belong to tf.gradients() you can find that tensorflow has done this gradient distribution part in a nice way.

While backtracking tf interact with graph , In the backword pass TF will meet different nodes Inside these nodes there are operations which we call (ops) matmal, softmax,relu, batch_normalization etc So what we tf does is it automatically load these ops in to the graph

This new node compose the partial derivative of the operations. get_gradient()

Let’s talk a bit about these newly added nodes

Inside these nodes we tf add 2 things 1. Derivative we calculated ealier ) 2.Also the inputs to the correspoding opp in the forward pass

So by the chain rule we can calculate

So this is so same like a backword API

So tensorflow always think about the order of the graph in order to do automatic differentiation

So as we know we need forward pass variables to calculate the gradients then we need to store intermidiate values also in tensors this can reduce the memory For many operations tf know how to calculate gradients and distribute them.

  • 1
    $\begingroup$ im using eager mode with no graph, no tf.function decorator, how does the auto differentation know the relation between tensors? $\endgroup$ – datdinhquoc Oct 1 '19 at 10:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.