# Questions tagged [automatic-differentiation]

The tag has no usage guidance.

14 questions
Filter by
Sorted by
Tagged with
39 views

### Difference between forward-mode and reverse-mode automatic differentiation?

I have difficulty grasping the difference between forward and reverse mode automatic differentiation. To understand this problem I have created a simple equation and broken this equation into small ...
• 2,031
1 vote
49 views

### Derivation of ELBO in ADVI Paper, Jacobian of Elliptical Transformation

I've been following the ELBO derivations in the paper Automatic Differentiation Variational Inference and have a few questions. With the model $p(x,\theta)$, they first transform $\theta$ so that it ...
• 11
856 views

### In GD-optimisation, if the gradient of the error function is w.r.t to the weights, isn't the target value dropped since it's a lone constant?

Suppose we have the absolute difference as an error function: $\mathit{loss}(w) = |m_x(w) - t|$ where $m_x$ is simply some model with input $x$ and weight setting $w$, and $t$ is the target value. In ...
• 619
31 views

• 143
2k views

### Vector Jacobian product in automatic differentiation

my questions is related to this post Higher Order of Vectorization in Backpropagation in Neural Network @shimao I don't really get the following claim (I know how the chain rule works and what is the ...
• 97
1 vote
132 views

### Reverse-Mode Automatic Differentiation with respect to a Matrix: How to "Matrix Multiply" 4D Tensors?

This is a follow up question I have on this excellent answer: https://stats.stackexchange.com/a/235758/307400. I will save me writing down any details about reverse-mode automatic differentiation, the ...
• 111
213 views

### Vector-Jacobian Product Computational Cost

The paper FFJORD: Free-form Continuous Dynamics for Scalable Reversible Generative Models presents a continuous-time flow as a generative model which uses Hutchinson's trace estimator to give an ...
• 31
1 vote
55 views

### automatic diffentiation (autograd): when the explicit definition of the gradient function is needed?

In Pytorch and similar machine learning software, the Autograd module computes the gradient of a function without needing to explicit declare the derivative of each single function which composes the ...
• 625
225 views

### Automatic differentiation for a function without representation

I have been studying AD for these days and I think I understand how it works, but all functions for which AD has been applied in the lectures I've studied are elementary in the mathematical sense, I ...
1 vote
97 views

### Auto Differentiation in Deep Learning Libraries

It is said that auto-diff is very efficient in generating the derivatives for backpropagation algorithms. The why is it that some of the most widely used deep learning libraries like Theano and ...
• 758
6k views

### What is an example use of Auto differentiation such as implemented in Tensorflow and why is it important?

I have a decent grasp of neural networks, back propagation and chain rule however I am struggling to understand auto differentiation. The below refer to auto differentiation outside the context of ...
• 335