After a little bit of reading on these two terms, I have the impression they are used for the same thing. So is there actually a difference between these two concepts, and if so, how are they different?
$\begingroup$
$\endgroup$
Add a comment
|
2 Answers
$\begingroup$
$\endgroup$
1
After some more reading I came to following conclusion:
- Invertible NN are just neural networks that represent bijective functions $f$.
Normalizing flows are invertible NN $f$ that also have a tractable determinant of the Jacobian $D_x f$ as well as a tractable inverse $f^{-1}$. This allows for following interpretation: Let $X \sim p_X, Z \sim p_Z$ be some random variable with $Z = f(X)$. Then $$p_X(x) = p_Z(f(x)) \det D_x f .$$ Because $f$ has a tractable inverse $f^{-1}$ we can therefore easily sample from one of the two distributions $p_X, p_Z$ by sampling from the other one and using the transformation above.
This could be applied in the following way (just as an example): We could train $f$ such that $p_X$ represents a distribution of images (e.g. represented by MNIST) and $p_Z$ a Gaussian. Then we can easily sample from the distribution of images by sampling $Z \sim p_Z$ (Gaussian) and just transforming it back to $X = f^{-1}(Z) \sim p_X$.
-
3$\begingroup$ Small note about normalizing flows: While their Jacobian has to be tractable for the flow to be useful, the inverse need not be. Take the planar flow (Rezende, 2015) for example that has no closed form inverse. Sampling from this flow is still possible as is likelihood calculation for those samples. $\endgroup$ Commented Aug 25, 2019 at 9:42
$\begingroup$
$\endgroup$
3
An invertible neural network is a general term used for any neural network that’s invertible. A flow neural network is a specific kind of invertible neural network. It’s just that it’s rather difficult to construct invertible neural networks, and flow neural networks offer an easy recipe.
-
$\begingroup$ Thanks a lot for the explanation! Can you maybe elaborate what is special about the flow networks, or what distinguishes them from the other invertible networks? $\endgroup$– flawrCommented Jun 26, 2019 at 15:21
-
$\begingroup$ @flawr: Take a look at the original papers, they are quite readable. Here's the one on NICE: arxiv.org/abs/1410.8516.pdf $\endgroup$– Alex R.Commented Jun 26, 2019 at 17:09
-
$\begingroup$ Thanks, that is what I was coming from - I also found the Glow and the RealNVP papers. While I think I do understand the ideas they present I just see that they are proposing certain architectures that allow for an easy inversion and computation of the corresponding determinants, but I don't see how "flow"-network is different from any other invertible networks. $\endgroup$– flawrCommented Jun 27, 2019 at 7:00