Questions tagged [automatic-differentiation]

Filter by
Sorted by
Tagged with
0 votes
0 answers
175 views

Meaning of vector Jacobian Product with matrix inputs

From my understanding of reverse-mode auto-differentiation, after the forward-pass computation graph has been constructed, gradients are passed from the loss down through the tree to the leaves. ...
Vityou's user avatar
  • 11
3 votes
1 answer
226 views

Natural gradients with Moore–Penrose inverse of the Fisher information matrix

I'd like to show you my rough sketch for scaling up natural gradients to deep neural networks that appears to be easy to automate just like automatic differentiation. I think there must be a flaw ...
all feedback welcome's user avatar
1 vote
0 answers
265 views

Difference between forward-mode and reverse-mode automatic differentiation?

I have difficulty grasping the difference between forward and reverse mode automatic differentiation. To understand this problem I have created a simple equation and broken this equation into small ...
Eka's user avatar
  • 2,211
3 votes
2 answers
195 views

Derivation of ELBO in ADVI Paper, Jacobian of Elliptical Transformation

I've been following the ELBO derivations in the paper Automatic Differentiation Variational Inference and have a few questions. With the model $p(x,\theta)$, they first transform $\theta$ so that it ...
James's user avatar
  • 33
7 votes
2 answers
1k views

In GD-optimisation, if the gradient of the error function is w.r.t to the weights, isn't the target value dropped since it's a lone constant?

Suppose we have the absolute difference as an error function: $\mathit{loss}(w) = |m_x(w) - t|$ where $m_x$ is simply some model with input $x$ and weight setting $w$, and $t$ is the target value. In ...
mesllo's user avatar
  • 679
3 votes
1 answer
87 views

How to know the number of dimensions of a Jacobian?

My question comes from a comment in this question Vector Jacobian product in automatic differentiation The question states... $$ t = Wz, \,\,\, z\in \mathbb{R}^{m\times 1}, t \in \mathbb{R}^{n \times ...
Joff's user avatar
  • 812
8 votes
1 answer
334 views

Mathematical notation for suppressing differentiation

Basic question Is the some existing mathematical notation to mean "treat this term as a constant when differentiating"? This would be the equivalent of ...
Dennis Prangle's user avatar
0 votes
0 answers
25 views

Computing the Jacobian $J_F$ with $F = h \circ f$

Let $$ f: \mathbb{R}^l \rightarrow{} \mathbb{R}^m\\[.7ex] h: \mathbb{R}^m \rightarrow{} \mathbb{R}^o$$ and let $$F = h \circ f \quad (F : \mathbb{R}^l \rightarrow{} \mathbb{R}^o)$$ I want to compute ...
lalaland's user avatar
  • 227
3 votes
0 answers
6k views

Vector Jacobian product in automatic differentiation

my questions is related to this post Higher Order of Vectorization in Backpropagation in Neural Network @shimao I don't really get the following claim (I know how the chain rule works and what is the ...
jdeJuan's user avatar
  • 127
1 vote
1 answer
463 views

Reverse-Mode Automatic Differentiation with respect to a Matrix: How to "Matrix Multiply" 4D Tensors?

This is a follow up question I have on this excellent answer: https://stats.stackexchange.com/a/235758/307400. I will save me writing down any details about reverse-mode automatic differentiation, the ...
cherrywoods's user avatar
3 votes
1 answer
667 views

Vector-Jacobian Product Computational Cost

The paper FFJORD: Free-form Continuous Dynamics for Scalable Reversible Generative Models presents a continuous-time flow as a generative model which uses Hutchinson's trace estimator to give an ...
Lashoun's user avatar
  • 33
1 vote
1 answer
105 views

automatic diffentiation (autograd): when the explicit definition of the gradient function is needed?

In Pytorch and similar machine learning software, the Autograd module computes the gradient of a function without needing to explicit declare the derivative of each single function which composes the ...
volperossa's user avatar
0 votes
2 answers
447 views

Automatic differentiation for a function without representation

I have been studying AD for these days and I think I understand how it works, but all functions for which AD has been applied in the lectures I've studied are elementary in the mathematical sense, I ...
Mads C's user avatar
  • 3
1 vote
0 answers
99 views

Auto Differentiation in Deep Learning Libraries

It is said that auto-diff is very efficient in generating the derivatives for backpropagation algorithms. The why is it that some of the most widely used deep learning libraries like Theano and ...
m1cro1ce's user avatar
  • 828
7 votes
1 answer
6k views

What is an example use of Auto differentiation such as implemented in Tensorflow and why is it important?

I have a decent grasp of neural networks, back propagation and chain rule however I am struggling to understand auto differentiation. The below refer to auto differentiation outside the context of ...
Greg's user avatar
  • 355
42 votes
1 answer
23k views

Step-by-step example of reverse-mode automatic differentiation

Not sure if this question belongs here, but it's closely related to gradient methods in optimization, which seems to be on-topic here. Anyway, feel free to migrate if you think some other community ...
ffriend's user avatar
  • 9,840