How to choose an appropriate variational distribution? I work in deep learning research and I am trying to learn how to use variational inference in order to approximate a posterior over the learned weights. 
I have looked extensively at Yarin Gal's papers  as a reference (https://arxiv.org/abs/1506.02158, https://arxiv.org/pdf/1506.02142). He approximates a mixture of Gaussians for each weight:
\begin{align*}
     q_\theta(W_1) = \prod_{q=1}^Q q_\theta(w_q), ~~~
     q_\theta(w_q) = p_1 \mathcal{N}(m_q, \sigma^2 I_K) + (1-p_1) \mathcal{N}(0, \sigma^2 I_K)
     \end{align*} 
By fixing some of the parameters to some constants, sampling from this distribution is equivalent to sampling from a Bernoulli distribution.
What I'm confused about is the rationale for choosing a variational distribution of this form? My gut feeling is that this was chosen specifically because it approximates Bernoulli drop-out and thus everyone gets Bayesian neural nets for free with no extra hassle. 
However, this leads me to the question, how does one choose an appropriate distribution given a certain problem? For instance, in my current work, I am aiming to model certain distributions that allow for clustering of the network weights, yet it makes no sense to me as how one would approach this from a variational perspective.
Any insights or good references that explain how to do this?
 A: In my opinion, the works you've linked are largely aimed at getting computationally efficient Bayesian neural networks (BNNs), by approximating variational BNNs via adding noise. For example, adding dropout regularization or multiplicative/Gaussian noise to the network then gives you a way to obtain BNN uncertainty (in these cases, predictive variance) at very little cost. Theoretically speaking, it converges to a variational BNN's uncertainty, if I recall correctly. See also Gal's thesis.

What I'm confused about is the rationale for choosing a variational distribution of this form? My gut feeling is that this was chosen specifically because it approximates Bernoulli drop-out and thus everyone gets Bayesian neural nets for free with no extra hassle.

Yes, exactly, your gut feeling is right (or at least aligns with mine). It is known, however, that this approach tends to underestimate the variance. Often these methods are not even variational (which is already an approximation itself), as the "optimization" is sort of implicit; rather, they are approximate variational methods in this sense.

However, this leads me to the question, how does one choose an appropriate distribution given a certain problem? For instance, in my current work, I am aiming to model certain distributions that allow for clustering of the network weights, yet it makes no sense to me as how one would approach this from a variational perspective.

I suppose it depends on the goal. For example, people tend to place a Laplace prior or a Horseshoe prior on the network weights, in order to enforce sparsity in a Bayesian way (e.g. [1]). Usually though, the variational distribution has a tradeoff between accuracy (matching the true posterior) and efficiency (not taking forever to calculate). 
The latter concern tends to dominate, causing people to use independent Gaussians across the weights.
In your case, perhaps your variational distribution could be a "linked" mixture model or something that forces all the parameters to change together across different weights.
I guess the issue is that normally the best variational distribution is the one with the most representational power that you can computationally afford; however, in your case, you are interested in manipulating the parameters of the model by the choice of variational distribution. My feeling is that people more often tend to use the prior for this, but I am not an expert. :)
