I have a neural network $x \mapsto f(x, \theta)$, and I can access predictions in my code with out = model(X). Imagine that I have a loss function $l(x,y) = (y-\frac{\partial f(x, \theta)}{\partial x}\cdot C)^2$ where $C$ is a constant vector. I think that I can approximate the derivative and write the code like this

eps = 0.001
pred = ... # compute the gradient with a formula like (f(x(1+eps))-f(x(1-eps)))/2x*eps
           # and make the dot product with C
loss = tf.math.squared_difference(y, pred) # loss for one data point

Is it possible to compute the gradient directly with respect to the input such that the output is still a function of $\theta$ and can be optimized by tensorflow? If so, what should I write in the line pred = ...?


  • 1
    $\begingroup$ Have you considered using PyTorch? Because this can be easily done using the torch.autograd package in PyTorch. $\endgroup$ – nagaK Aug 29 '20 at 18:59
  • $\begingroup$ Thanks, I will look at it $\endgroup$ – Hugo Laurençon Aug 29 '20 at 19:27

The best way to picture how this works is that automatic differentiation can be implemented by augmenting your existing computational graph with a few extra nodes / edges (which connect to your "forward comptuation") and compute the gradients. There's nothing which distinguishes these new nodes from your original forward computation (the graph itself is not aware of notions of "forward" versus "backward" pass), so it's completely possible to apply AD again, to get higher order derivatives.

Generally speaking, the backward "augmentation" of the graph is usually abstracted away from the end-user, but most machine learning frameworks such as tensorflow and pytorch do expose the relevant utilities, for example tf.gradients and torch.autograd.grad.

  • $\begingroup$ Thank you for your explanation $\endgroup$ – Hugo Laurençon Aug 29 '20 at 20:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.