# Variational inference with discrete variational parameters

Typically Variational Inference relies on taking gradient steps on KL divergence between the variational and true posterior, or on the ELBO. This does not seem valid when variational parameters are discrete (since gradients wrt those arguments are not defined). How can we perform VI on discrete variational parameters?

Well, in general this is an instance of a discrete optimization problem, and in general there are no methods more efficient than brute-force search over all possible values of these parameters. In some special cases though, one might use problem's structure to be much better. This is studied by the field of Combinatorial Optimization. In some cases when efficient exact solutions are unknown, there exist good approximate ones.

There are practical heuristics such as Evolutionary algorithms which are not guaranteed to be better than an exhaustive search, but might lead to a decent solution (in a short amount of time).

However, design of a variational approximation in VI is arbitrary, and it makes sense to choose its family so as to make the optimization process efficient and tractable. In practice, variational approximations are almost always parametrized by continuous parameters and thus allow gradient-based optimization.

• how about using continuous approximations via the Gumbel-Softmax trick or concrete distributions?
– Dion
Commented Sep 27, 2019 at 23:38
• @DionysisM, Gumbel-Softmax solves an unrelated problem. It assumes you have discrete random variables parametrized by continuous parameters and it relaxes these discrete r.v. into continuous ones. Crucially, parameters are continuous all along. Commented Sep 28, 2019 at 13:26
• One could attempt using Gumbel-Softmax for discrete parameters, but the problem here is that discrete parameters are not random variables, they are deterministic values, and thus the particular technique of G-S does not bring anything. You could consider other relaxations though – this is also a part of approximate discrete optimization. Commented Sep 28, 2019 at 13:28

It is correct that typical variational inference methods model the surrogate distribution $$q_{\phi}(z \vert x)$$ as a continuous latent space. A challenge of the following approach is dealing with posterior collapse. Provided with a trained powerful neural network decoder, i.e. universal function approximator, only a subset of the latent variables $$z_{i} \in z$$ will be exploited for decoding. This causes the variational distribution to replicate the prior, and results in difficulties for the decoding network to leverage the information contained within the latent space.

In particular, Vector Quantised Variational AutoEncoder (VQ-VAE) was a method developed to deal with the challenges of posterior collapse by leveraging instead a discrete latent representation. In more detail, the posterior and prior distributions are categorical. The data points are selected from these distributions using an index latent embedding table $$e \in \mathbb{R}^{K \times D}$$ where $$K$$ is the size of the discrete latent space (K-way categorical) and $$D$$ the dimensionality of each latent embedding vector $$e_{i} \in \mathbb{R}^{D}, i \in 1,2,\ldots,K$$. The posterior categorical distribution is defined as

$$q(z = k \vert x) = \begin{cases} 1 &\text{if } \text{for k} \text{=} \text{argmin}_{j} \Vert z_e(x) - e_j \Vert_2 \end{cases}$$

$$q(z = k \vert x) = \begin{cases} 0 &\text{} \text{otherwise} \end{cases}$$

Additionally, a follow-up publication VQ-VAE2 to the original was published for large scale image generation, improving the coherence and fidelity of image generation.

The following excerpts are taken from my book on variational inference. Learn more on the topic by visiting https://www.thevariationalbook.com/