# Questions tagged [automatic-differentiation]

The tag has no usage guidance.

16 questions
Filter by
Sorted by
Tagged with
175 views

### Meaning of vector Jacobian Product with matrix inputs

From my understanding of reverse-mode auto-differentiation, after the forward-pass computation graph has been constructed, gradients are passed from the loss down through the tree to the leaves. ...
226 views

### Natural gradients with Moore–Penrose inverse of the Fisher information matrix

I'd like to show you my rough sketch for scaling up natural gradients to deep neural networks that appears to be easy to automate just like automatic differentiation. I think there must be a flaw ...
1 vote
265 views

### Difference between forward-mode and reverse-mode automatic differentiation?

I have difficulty grasping the difference between forward and reverse mode automatic differentiation. To understand this problem I have created a simple equation and broken this equation into small ...
195 views

### Derivation of ELBO in ADVI Paper, Jacobian of Elliptical Transformation

I've been following the ELBO derivations in the paper Automatic Differentiation Variational Inference and have a few questions. With the model $p(x,\theta)$, they first transform $\theta$ so that it ...
1k views

### In GD-optimisation, if the gradient of the error function is w.r.t to the weights, isn't the target value dropped since it's a lone constant?

Suppose we have the absolute difference as an error function: $\mathit{loss}(w) = |m_x(w) - t|$ where $m_x$ is simply some model with input $x$ and weight setting $w$, and $t$ is the target value. In ...
87 views

6k views

### Vector Jacobian product in automatic differentiation

my questions is related to this post Higher Order of Vectorization in Backpropagation in Neural Network @shimao I don't really get the following claim (I know how the chain rule works and what is the ...
1 vote
463 views

### Reverse-Mode Automatic Differentiation with respect to a Matrix: How to "Matrix Multiply" 4D Tensors?

This is a follow up question I have on this excellent answer: https://stats.stackexchange.com/a/235758/307400. I will save me writing down any details about reverse-mode automatic differentiation, the ...
667 views

### Vector-Jacobian Product Computational Cost

The paper FFJORD: Free-form Continuous Dynamics for Scalable Reversible Generative Models presents a continuous-time flow as a generative model which uses Hutchinson's trace estimator to give an ...
1 vote
105 views

### automatic diffentiation (autograd): when the explicit definition of the gradient function is needed?

In Pytorch and similar machine learning software, the Autograd module computes the gradient of a function without needing to explicit declare the derivative of each single function which composes the ...
447 views

### Automatic differentiation for a function without representation

I have been studying AD for these days and I think I understand how it works, but all functions for which AD has been applied in the lectures I've studied are elementary in the mathematical sense, I ...
1 vote
99 views

### Auto Differentiation in Deep Learning Libraries

It is said that auto-diff is very efficient in generating the derivatives for backpropagation algorithms. The why is it that some of the most widely used deep learning libraries like Theano and ...