# Why aren't Normalizing Flows suitable for Discrete Distributions?

I am currently trying to understand why normalizing flows are not applicable to discrete distributions (a quick primer on NF can be found here). The assumptions on the transformation f between the probability distributions are:

1. f must be an invertible function
2. f must be a smooth function

Assume I want to learn a normalizing flow between a Poisson and a Normal distribution.
If I discretize my Normal distribution, both have an infinite support, i.e., the same number of support elements and hence, I can find an invertible mapping between them (another option would be to consider only the first n natural numbers for the Poisson distribution and also select n elements from the Normal distribution).

Moreover, when I am training a Normalizing Flow on my computer, I never have continuous data - instead, I have discrete samples from my distribution. So, where is the issue with taking these samples from a discrete distribution now? Unfortunately, I cannot see the difference to the case where I am trying to map between two Normal distributions: if I sample from a Normal distribution I have discrete samples as well.

An NF is essentially just a change of variable. If you want to change a density supported on a discrete set, to a density supported on a continuous set, the corresponding transformation is bound to degenerate (which manifests as a Jacobian determinant of infinity or 0, depending which direction of the flow you consider). Actually, even training a NF for data living on a low-dimensional manifold will lead to the same problem, thus this degeneracy problem is not only limited to discrete-continuous case, see e.g. https://proceedings.neurips.cc/paper/2021/file/4c07fe24771249c343e70c32289c1192-Paper.pdf

The fundamental idea is to use the change of variables formula, which says that if you have a continuous density, $$p_z$$ and a bijection that is also differentiable, $$u = \phi(z), z = \phi^{-1}(u)$$, then we get a new density in the variable $$u$$ as follows: $$p_u(u):= p_z(\phi^{-1}(u)) |\det D_u \phi^{-1}(u)| = \frac{p_z(z)}{|\det D_z \phi(z)|}$$ Here $$D_u$$ is the Jacobian (derivative matrix) in variable $$u$$, similarly for $$D_z$$. This is easy to prove by just writing $$1 = \int p_z dz = \int p_z(\phi^{-1}(u)) |\det D_u \phi^{-1}(u)|$$ (change variables in the integral) and define $$p_u$$ to be the integrand on the right. For any measurable set $$S$$, $$P_z(S) = P_u(\phi(S)),$$ which is proved similarly.

You mention

when I am training a Normalizing Flow on my computer, I never have continuous data - instead, I have discrete samples from my distribution.

if your data $$\{x_i\}$$ is a sample from a continuous density (e.g., if one samples from $$N(0,I)$$ we obtain many discrete data points in $$\mathbb{R}^n$$, but the density is over the continuous sample space $$\mathbb{R}^n$$), then there is no problem. The issue arises when the sample space is discrete e.g., $$\{a, b, c, d\}$$.

When moving to a categorical/discrete sample space we have probability mass functions, and the notion of the derivative is unclear.

There are works to adapt the normalizing flows to the discrete scenario:

1. This paper pursues a change of variables formula for discrete distributions and uses the analogous normalizing flow setup with it.
2. This paper has the first layer embed the categorical input data to $$\mathbb{R}^k$$ using variational inference and the last $$n-1$$ layers a continuous-data (usual, original) normalizing flow. The whole network is trained using an ELBO method.