Why aren't Normalizing Flows suitable for Discrete Distributions?

Question

I am currently trying to understand why normalizing flows are not applicable to discrete distributions (a quick primer on NF can be found here). The assumptions on the transformation f between the probability distributions are:

1. f must be an invertible function
2. f must be a smooth function

Assume I want to learn a normalizing flow between a Poisson and a Normal distribution.
If I discretize my Normal distribution, both have an infinite support, i.e., the same number of support elements and hence, I can find an invertible mapping between them (another option would be to consider only the first n natural numbers for the Poisson distribution and also select n elements from the Normal distribution).

Moreover, when I am training a Normalizing Flow on my computer, I never have continuous data - instead, I have discrete samples from my distribution. So, where is the issue with taking these samples from a discrete distribution now? Unfortunately, I cannot see the difference to the case where I am trying to map between two Normal distributions: if I sample from a Normal distribution I have discrete samples as well.

Thanks in advance!

Answer 1

An NF is essentially just a change of variable. If you want to change a density supported on a discrete set, to a density supported on a continuous set, the corresponding transformation is bound to degenerate (which manifests as a Jacobian determinant of infinity or 0, depending which direction of the flow you consider). Actually, even training a NF for data living on a low-dimensional manifold will lead to the same problem, thus this degeneracy problem is not only limited to discrete-continuous case, see e.g. https://proceedings.neurips.cc/paper/2021/file/4c07fe24771249c343e70c32289c1192-Paper.pdf

Answer 2

The fundamental idea is to use the change of variables formula, which says that if you have a continuous density, $p_z$ and a bijection that is also differentiable, $u = \phi(z), z = \phi^{-1}(u)$, then we get a new density in the variable $u$ as follows: $$p_u(u):= p_z(\phi^{-1}(u)) |\det D_u \phi^{-1}(u)| = \frac{p_z(z)}{|\det D_z \phi(z)|}$$ Here $D_u$ is the Jacobian (derivative matrix) in variable $u$, similarly for $D_z$. This is easy to prove by just writing $1 = \int p_z dz = \int p_z(\phi^{-1}(u)) |\det D_u \phi^{-1}(u)|$ (change variables in the integral) and define $p_u$ to be the integrand on the right. For any measurable set $S$, $P_z(S) = P_u(\phi(S)),$ which is proved similarly.

You mention

when I am training a Normalizing Flow on my computer, I never have continuous data - instead, I have discrete samples from my distribution.

if your data $\{x_i\}$ is a sample from a continuous density (e.g., if one samples from $N(0,I)$ we obtain many discrete data points in $\mathbb{R}^n$, but the density is over the continuous sample space $\mathbb{R}^n$), then there is no problem. The issue arises when the sample space is discrete e.g., $\{a, b, c, d\}$.

When moving to a categorical/discrete sample space we have probability mass functions, and the notion of the derivative is unclear.

There are works to adapt the normalizing flows to the discrete scenario:

1. This paper pursues a change of variables formula for discrete distributions and uses the analogous normalizing flow setup with it.
2. This paper has the first layer embed the categorical input data to $\mathbb{R}^k$ using variational inference and the last $n-1$ layers a continuous-data (usual, original) normalizing flow. The whole network is trained using an ELBO method.