# How does Tensorflow tf.train.Optimizer compute gradients?

I'm following the Tensorflow mnist tutorial (https://github.com/tensorflow/tensorflow/blob/master/tensorflow/examples/tutorials/mnist/mnist_softmax.py).

The tutorial uses tf.train.Optimizer.minimize (specifically tf.train.GradientDescentOptimizer). I don't see any arguments being passed in anywhere to define gradients.

Is Tensor flow using numerical differentiation by default?

Is there a way to pass in gradients like you can with scipy.optimize.minimize?

It's not numerical differentiation, it's automatic differentiation. This is one of the main reasons for tensorflow's existence: by specifying operations in a tensorflow graph (with operations on Tensors and so on), it can automatically follow the chain rule through the graph and, since it knows the derivatives of each individual operation you specify, it can combine them automatically.

If for some reason you want to override that piecewise, it's possible with gradient_override_map.

• Isn't automatic differentiation using numerical differentitation? – Aerin Jun 21 '17 at 21:39
• @BYOR No; check out the Wikipedia link above. What tensorflow does is actually somewhere in between "real" reverse-mode autodiff and symbolic differentiation, though. – Dougal Jun 21 '17 at 21:42
• @Dougal im using eager mode with no graph, no tf.function decorator, how does the auto differentation know the relation between tensors? – datdinhquoc Oct 1 '19 at 10:19
• @datdinhquoc It's the same fundamental algorithm, just a little more complicated of an implementation. – Dougal Oct 1 '19 at 12:22

It uses automatic differentiation. Where it uses chain rule and go backword in the graph assigning gradients.

Let’s say we have a tensor C This tensor C has made after series of operations Let’s say by adding , multiplying , going through some nonlinearity etc

So if this C depends on some set of tensors called Xk , We need to get the gradients

Tensorflow always track the path of operations. I mean the sequential behavior of the nodes and how data flow between them. That is done by the graph

If we need to get the derivatices of the cost w.r.t X inputs what this will first does is it load the path from x-input to the cost by extending the graph.

Then it start in the rivers order. Then distribute the gradients with chain rule. (Same as backpropagation)

Any way if you read the source codes belong to tf.gradients() you can find that tensorflow has done this gradient distribution part in a nice way.

While backtracking tf interact with graph , In the backword pass TF will meet different nodes Inside these nodes there are operations which we call (ops) matmal, softmax,relu, batch_normalization etc So what we tf does is it automatically load these ops in to the graph

This new node compose the partial derivative of the operations. get_gradient()