I am currently trying to understand why normalizing flows are not applicable to discrete distributions (a quick primer on NF can be found here). The assumptions on the transformation f between the probability distributions are:

  1. f must be an invertible function
  2. f must be a smooth function

Assume I want to learn a normalizing flow between a Poisson and a Normal distribution.
If I discretize my Normal distribution, both have an infinite support, i.e., the same number of support elements and hence, I can find an invertible mapping between them (another option would be to consider only the first n natural numbers for the Poisson distribution and also select n elements from the Normal distribution).

Moreover, when I am training a Normalizing Flow on my computer, I never have continuous data - instead, I have discrete samples from my distribution. So, where is the issue with taking these samples from a discrete distribution now? Unfortunately, I cannot see the difference to the case where I am trying to map between two Normal distributions: if I sample from a Normal distribution I have discrete samples as well.

Thanks in advance!


2 Answers 2


An NF is essentially just a change of variable. If you want to change a density supported on a discrete set, to a density supported on a continuous set, the corresponding transformation is bound to degenerate (which manifests as a Jacobian determinant of infinity or 0, depending which direction of the flow you consider). Actually, even training a NF for data living on a low-dimensional manifold will lead to the same problem, thus this degeneracy problem is not only limited to discrete-continuous case, see e.g. https://proceedings.neurips.cc/paper/2021/file/4c07fe24771249c343e70c32289c1192-Paper.pdf


The fundamental idea is to use the change of variables formula, which says that if you have a continuous density, $p_z$ and a bijection that is also differentiable, $u = \phi(z), z = \phi^{-1}(u)$, then we get a new density in the variable $u$ as follows: $$p_u(u):= p_z(\phi^{-1}(u)) |\det D_u \phi^{-1}(u)| = \frac{p_z(z)}{|\det D_z \phi(z)|}$$ Here $D_u$ is the Jacobian (derivative matrix) in variable $u$, similarly for $D_z$. This is easy to prove by just writing $1 = \int p_z dz = \int p_z(\phi^{-1}(u)) |\det D_u \phi^{-1}(u)|$ (change variables in the integral) and define $p_u$ to be the integrand on the right. For any measurable set $S$, $P_z(S) = P_u(\phi(S)),$ which is proved similarly.

You mention

when I am training a Normalizing Flow on my computer, I never have continuous data - instead, I have discrete samples from my distribution.

if your data $\{x_i\}$ is a sample from a continuous density (e.g., if one samples from $N(0,I)$ we obtain many discrete data points in $\mathbb{R}^n$, but the density is over the continuous sample space $\mathbb{R}^n$), then there is no problem. The issue arises when the sample space is discrete e.g., $\{a, b, c, d\}$.

When moving to a categorical/discrete sample space we have probability mass functions, and the notion of the derivative is unclear.

There are works to adapt the normalizing flows to the discrete scenario:

  1. This paper pursues a change of variables formula for discrete distributions and uses the analogous normalizing flow setup with it.
  2. This paper has the first layer embed the categorical input data to $\mathbb{R}^k$ using variational inference and the last $n-1$ layers a continuous-data (usual, original) normalizing flow. The whole network is trained using an ELBO method.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.