How does Tensorflow `tf.train.Optimizer` compute gradients? I'm following the Tensorflow mnist tutorial (https://github.com/tensorflow/tensorflow/blob/master/tensorflow/examples/tutorials/mnist/mnist_softmax.py).
The tutorial uses tf.train.Optimizer.minimize (specifically tf.train.GradientDescentOptimizer). I don't see any arguments being passed in anywhere to define gradients.
Is Tensor flow using numerical differentiation by default?
Is there a way to pass in gradients like you can with scipy.optimize.minimize?
 A: It's not numerical differentiation, it's automatic differentiation. This is one of the main reasons for tensorflow's existence: by specifying operations in a tensorflow graph (with operations on Tensors and so on), it can automatically follow the chain rule through the graph and, since it knows the derivatives of each individual operation you specify, it can combine them automatically.
If for some reason you want to override that piecewise, it's possible with gradient_override_map. 
A: It uses automatic differentiation. Where it uses chain rule and go backword in the graph assigning gradients. 
Let’s say we have a tensor C
This tensor C  has made after series of operations 
Let’s say by adding , multiplying , going through some nonlinearity etc  
So if this C depends on some set of tensors called Xk ,
We need to get the gradients 
Tensorflow always track the path of operations. I mean the sequential behavior of the nodes and how data flow between them. That is done by the graph 

If we need to get the derivatices of the cost w.r.t X inputs what this will first does is it load the path from x-input to the cost by extending the graph.
Then it start in the rivers order. Then distribute the gradients with chain rule. (Same as backpropagation)
Any way if you read the source codes belong to tf.gradients() you can find that tensorflow has done this gradient distribution part in a nice way.  
While backtracking tf interact with graph ,
In the backword pass TF will meet different nodes 
Inside these nodes there are operations which we call (ops) 
matmal, softmax,relu, batch_normalization etc 
So what we tf does is it automatically load these ops  in to the graph 
This new node compose the partial derivative of the operations. get_gradient()
Let’s talk a bit about these newly added nodes 
Inside these nodes we tf add 2 things 
1. Derivative we calculated ealier )
2.Also the inputs to the correspoding opp in the forward pass 
So by the chain rule we can calculate  
So this is so same like a backword API 
So tensorflow always think about the order of the graph in order to do automatic differentiation 
So as we know we need forward pass variables to calculate the gradients 
then we need to store intermidiate values also in tensors this can reduce the memory 
For many operations tf know how to calculate gradients and distribute them.
