It uses automatic differentiation. Where it uses chain rule and go backword in the graph assigning gradients.
Let’s say we have a tensor C
This tensor C has made after series of operations
Let’s say by adding , multiplying , going through some nonlinearity etc
So if this C depends on some set of tensors called Xk ,
We need to get the gradients
Tensorflow always track the path of operations. I mean the sequential behavior of the nodes and how data flow between them. That is done by the graph
If we need to get the derivatices of the cost w.r.t X inputs what this will first does is it load the path from x-input to the cost by extending the graph.
Then it start in the rivers order. Then distribute the gradients with chain rule. (Same as backpropagation)
Any way if you read the source codes belong to tf.gradients() you can find that tensorflow has done this gradient distribution part in a nice way.
While backtracking tf interact with graph ,
In the backword pass TF will meet different nodes
Inside these nodes there are operations which we call (ops)
matmal, softmax,relu, batch_normalization etc
So what we tf does is it automatically load these ops in to the graph
This new node compose the partial derivative of the operations. get_gradient()
Let’s talk a bit about these newly added nodes
Inside these nodes we tf add 2 things
1. Derivative we calculated ealier )
2.Also the inputs to the correspoding opp in the forward pass
So by the chain rule we can calculate
So this is so same like a backword API
So tensorflow always think about the order of the graph in order to do automatic differentiation
So as we know we need forward pass variables to calculate the gradients
then we need to store intermidiate values also in tensors this can reduce the memory
For many operations tf know how to calculate gradients and distribute them.