I have a neural network $x \mapsto f(x, \theta)$, and I can access predictions in my code with out = model(X). Imagine that I have a loss function $l(x,y) = (y-\frac{\partial f(x, \theta)}{\partial x}\cdot C)^2$ where $C$ is a constant vector. I think that I can approximate the derivative and write the code like this

eps = 0.001
pred = ... # compute the gradient with a formula like (f(x(1+eps))-f(x(1-eps)))/2x*eps
           # and make the dot product with C
loss = tf.math.squared_difference(y, pred) # loss for one data point

Is it possible to compute the gradient directly with respect to the input such that the output is still a function of $\theta$ and can be optimized by tensorflow? If so, what should I write in the line pred = ...?


  • 1
    $\begingroup$ Have you considered using PyTorch? Because this can be easily done using the torch.autograd package in PyTorch. $\endgroup$
    – nagaK
    Aug 29, 2020 at 18:59
  • $\begingroup$ Thanks, I will look at it $\endgroup$ Aug 29, 2020 at 19:27

1 Answer 1


The best way to picture how this works is that automatic differentiation can be implemented by augmenting your existing computational graph with a few extra nodes / edges (which connect to your "forward comptuation") and compute the gradients. There's nothing which distinguishes these new nodes from your original forward computation (the graph itself is not aware of notions of "forward" versus "backward" pass), so it's completely possible to apply AD again, to get higher order derivatives.

Generally speaking, the backward "augmentation" of the graph is usually abstracted away from the end-user, but most machine learning frameworks such as tensorflow and pytorch do expose the relevant utilities, for example tf.gradients and torch.autograd.grad.

  • $\begingroup$ Thank you for your explanation $\endgroup$ Aug 29, 2020 at 20:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.