What are the advantages of normalizing flow over VAEs with deep latent gaussian models for inference?

Question

I am reading the normalizing flow paper and am a bit confused.

It seems that being able to model complex (correlated?) posterior is one of the advantages of the proposed approach (Section 2.3, last paragraph). But for a deep latent Gaussian model with diagonal covariance, i.e. $$z^{(1)}\sim N(\mu^{(1)}(x),\mathrm{diag}(\sigma^{(1)}(x))),\\ z^{(2)}\sim N(\mu^{(2)}(z^{(1)}),\mathrm{diag}(\sigma^{(2)}(z^{(1)}))),$$ $P(z^{(2)}|x)$ can also be arbitrarily complex. So why is normalizing flow considered to be more expressive? What did I get wrong?

Answer 1

So the answer lies in the PhD thesis of Durk Kingma. In his thesis he has mentioned that

The framework of normalizing flows [Rezende and Mohamed, 2015] provides an attractive approach for parameterizing flexible approximate posterior distributions in the VAE framework

The term "flexible approximation" is useful so that the model adapts to the data, which reduces the distance between real distribution and model distribution. But the problem which is faced that the framework of normalizing flows is that it doesn't scale well in high dimensional latent spaces. To overcome this issue

[Kingma et al., 2016] we propose inverse autoregressive flows, a flexible class of posterior distributions based on normalizing flows, allowing inference of highly non-Gaussian posterior distributions over high-dimensional latent spaces.

All the information for the above answer was taken from PhD thesis if Durk Kingma, the link of the same can be found here: http://dpkingma.com