12
votes
Accepted
Difference between invertible NN and flow-based NN
After some more reading I came to following conclusion:
Invertible NN are just neural networks that represent bijective functions $f$.
Normalizing flows are invertible NN $f$ that also have a ...
- 583
7
votes
Accepted
Normalizing flow training
Overview.
You're familiar with MLE, which is a good starting point. We have a parametric model whose parameters $\theta$ we seek to optimize, in order to maximize the likelihood of our model $L(\theta ...
- 8,437
4
votes
Accepted
What are the advantages of normalizing flow over VAEs with deep latent gaussian models for inference?
So the answer lies in the PhD thesis of Durk Kingma. In his thesis he has mentioned that
The framework of normalizing flows [Rezende and Mohamed, 2015] provides
an attractive approach for ...
- 166
3
votes
Accepted
Issues with GAN and VAE models
They're some sort of generative models, learning the PDF so that they can sample from it. This is achieved by having random latent representations and mapping them to the relevant domain. So, you can ...
- 56.4k
3
votes
Accepted
Why aren't Normalizing Flows suitable for Discrete Distributions?
An NF is essentially just a change of variable. If you want to change a density supported on a discrete set, to a density supported on a continuous set, the corresponding transformation is bound to ...
- 46
3
votes
Difference between invertible NN and flow-based NN
An invertible neural network is a general term used for any neural network that’s invertible. A flow neural network is a specific kind of invertible neural network. It’s just that it’s rather ...
- 13.8k
3
votes
Accepted
Vector-Jacobian Product Computational Cost
This is a well known result from automatic differentiation literature. Specifically, the result is that reverse mode differentiation can
calculate the gradient $\frac{\partial \hat{f}} {\partial \...
- 208
3
votes
Accepted
Planar Flow in Normalizing Flows
For every $z,$ notice that the displacement from $z$ to its destination $f(z),$ given by $f(z)-z,$ is a multiple of the fixed vector $u.$ Thus, if you were to diagram the effect of $f$ by drawing ...
- 316k
2
votes
Planar Flow in Normalizing Flows
The equation
$$
\mathbf{w}^T\mathbf{z_1}+b=0
$$
defines a (hyper)plane. The vector $\mathbf{w}$ is the normal vector. For a refresher on multivariable calculus, see here.
So what happens if you have ...
- 20.2k
2
votes
Normalizing Flows KL divergence equivalency
The answer to your first question follows from the fact that the Kullback-Leibler divergence is, under mild conditions, invariant under transformations. This is straightforward and is shown in the ...
- 10.5k
1
vote
Accepted
Normalizing Flows Invertibility
I think it is important to recognize again that the so called coupling layers are splitted into two. One part directly passes to next layer without any modifications (i.e. $\pmb x_{1:d}$). That's why ...
- 36
1
vote
Normalizing Flows KL divergence equivalency
In short
The Kullback-Leibler divergence is the expectation value of the log-odds of two distributions
$$D_{KL}(A || B) = \textbf{E}_A\left[\log \left(\frac{P_A(x)}{P_B(x)} \right) \right]$$
or for ...
- 71.9k
1
vote
Accepted
Which parameters are updated in VAE with normalizing flow?
You optimize the loss with respect to $\theta$ and $\phi$—which includes the parameters of the decoder, the encoder, and the flow model.
The source code in the blog post you've linked to answers the ...
- 8,437
1
vote
Accepted
What is multi-scale architecture?
CNNs are typically constructed by stacking convolutional layers on top of each other, with each convolutional layer taking in the previous feature map and producing a successive feature map. The ...
- 25.6k
1
vote
Inference in Normalizing Flow model: NICE(non linear independent components estimation)
Since $f$ is bijective, you could also implement its inverse $f^{-1}(h) = x$. For example, to invert the first layer of the network $h_{I_1}^1 = x_{I_1}, h_{I_2}^1 = x_{I_2} + M(x_{I_1})$, you can ...
- 25.6k
Only top scored, non community-wiki answers of a minimum length are eligible