# How to choose an appropriate variational distribution?

I work in deep learning research and I am trying to learn how to use variational inference in order to approximate a posterior over the learned weights.

I have looked extensively at Yarin Gal's papers as a reference (https://arxiv.org/abs/1506.02158, https://arxiv.org/pdf/1506.02142). He approximates a mixture of Gaussians for each weight:

\begin{align*} q_\theta(W_1) = \prod_{q=1}^Q q_\theta(w_q), ~~~ q_\theta(w_q) = p_1 \mathcal{N}(m_q, \sigma^2 I_K) + (1-p_1) \mathcal{N}(0, \sigma^2 I_K) \end{align*}

By fixing some of the parameters to some constants, sampling from this distribution is equivalent to sampling from a Bernoulli distribution.

What I'm confused about is the rationale for choosing a variational distribution of this form? My gut feeling is that this was chosen specifically because it approximates Bernoulli drop-out and thus everyone gets Bayesian neural nets for free with no extra hassle.

However, this leads me to the question, how does one choose an appropriate distribution given a certain problem? For instance, in my current work, I am aiming to model certain distributions that allow for clustering of the network weights, yet it makes no sense to me as how one would approach this from a variational perspective.

Any insights or good references that explain how to do this?

In my opinion, the works you've linked are largely aimed at getting computationally efficient Bayesian neural networks (BNNs), by approximating variational BNNs via adding noise. For example, adding dropout regularization or multiplicative/Gaussian noise to the network then gives you a way to obtain BNN uncertainty (in these cases, predictive variance) at very little cost. Theoretically speaking, it converges to a variational BNN's uncertainty, if I recall correctly. See also Gal's thesis.

What I'm confused about is the rationale for choosing a variational distribution of this form? My gut feeling is that this was chosen specifically because it approximates Bernoulli drop-out and thus everyone gets Bayesian neural nets for free with no extra hassle.

Yes, exactly, your gut feeling is right (or at least aligns with mine). It is known, however, that this approach tends to underestimate the variance. Often these methods are not even variational (which is already an approximation itself), as the "optimization" is sort of implicit; rather, they are approximate variational methods in this sense.

However, this leads me to the question, how does one choose an appropriate distribution given a certain problem? For instance, in my current work, I am aiming to model certain distributions that allow for clustering of the network weights, yet it makes no sense to me as how one would approach this from a variational perspective.

I suppose it depends on the goal. For example, people tend to place a Laplace prior or a Horseshoe prior on the network weights, in order to enforce sparsity in a Bayesian way (e.g. ). Usually though, the variational distribution has a tradeoff between accuracy (matching the true posterior) and efficiency (not taking forever to calculate). The latter concern tends to dominate, causing people to use independent Gaussians across the weights. In your case, perhaps your variational distribution could be a "linked" mixture model or something that forces all the parameters to change together across different weights. I guess the issue is that normally the best variational distribution is the one with the most representational power that you can computationally afford; however, in your case, you are interested in manipulating the parameters of the model by the choice of variational distribution. My feeling is that people more often tend to use the prior for this, but I am not an expert. :)

• Thanks for the answer. Do you have any references on the fact "that this approach tends to underestimate the variance"? I've always heard the maximum-likelihood approaches to underestimate it, and I've seen this empirically but it would be nice to see some theoretical justification! – user2037067 Feb 5 '19 at 15:13
• 1. "the variational distribution has a tradeoff between accuracy (matching the true posterior)" The thing is, we have no clue about the distribution of the true posterior so how can we effectively choose a VD based on this? I like this paper (arxiv.org/pdf/1802.10501.pdf) which shows that in classification, the softmax likelihood can be seen as a Categorical distribution so a modelling a Dirichlet prior on this makes a lot of sense as it is its conjugate prior. 2. "and efficiency...use independent Gaussians across the weights". This is the mean field approximation? – user2037067 Feb 5 '19 at 15:27
• @user2037067 Regarding the variance underestimation, good question. Yarin Gal's papers & thesis might talk a bit about it. Maybe looking at how the convergence of the (co)variance wrt sample size works might give an answer. I think most discussions I've seen mention it empirically. I might also conjecture (especially for the deep approximate VBNNs e.g., based on dropout) that deep networks are often miscalibrated to be over-confident, which is possibly due to loss functions preferring high confidence answers, and that this leads to lower empirical variance measures perhaps. – user3658307 Feb 5 '19 at 17:59
• @user2037067 1. In my opinion, if you have no clue, choose the most powerful distribution that you can computationally afford. Independent Gaussians are weak but cheap. A little stronger allows correlations within layers, but not between them. Or you can allow complete covariance matrices (rather costly). Another way is to use mixture models to allow multi-modality, instead of having the model capture correlations better, for example. 2. Yes. PS: sorry I don't follow this field currently. Maybe people in a different SE would know more. – user3658307 Feb 5 '19 at 18:03
• @user2037067 I just saw Radial and Directional Posteriors for Bayesian Neural Networks, which might be a nice example for you in how to design BNN posterior parameterizations. – user3658307 Feb 10 '19 at 20:51