Questions tagged [variational-bayes]

Variational Bayesian methods approximate intractable integrals found in Bayesian inference and machine learning. Primarily, these methods serve one of two purposes: Approximating the posterior distribution, or bounding the marginal likelihood of observed data.

Filter by
Sorted by
Tagged with
58
votes
7answers
33k views

How does the reparameterization trick for VAEs work and why is it important?

How does the reparameterization trick for variational autoencoders (VAE) work? Is there an intuitive and easy explanation without simplifying the underlying math? And why do we need the 'trick'?
37
votes
1answer
6k views

Variational inference versus MCMC: when to choose one over the other?

I think I get the general idea of both VI and MCMC including the various flavors of MCMC like Gibbs sampling, Metropolis Hastings etc. This paper provides a wonderful exposition of both methods. I ...
26
votes
2answers
9k views

how to weight KLD loss vs reconstruction loss in variational auto-encoder

in nearly all code examples I've seen of a VAE, the loss functions are defined as follows (this is tensorflow code, but I've seen similar for theano, torch etc. It's also for a convnet, but that's ...
26
votes
1answer
6k views

Relation between variational Bayes and EM

I read somewhere that Variational Bayes method is a generalization of the EM algorithm. Indeed, the iterative parts of the algorithms are very similar. In order to test whether the EM algorithm is a ...
24
votes
1answer
4k views

What are variational autoencoders and to what learning tasks are they used?

As per this and this answer, autoencoders seem to be a technique that uses neural networks for dimension reduction. I would like to additionally know what is a variational autoencoder (its main ...
19
votes
4answers
7k views

When should I use a variational autoencoder as opposed to an autoencoder?

I understand the basic structure of variational autoencoder and normal (deterministic) autoencoder and the math behind them, but when and why would I prefer one type of autoencoder to the other? All I ...
12
votes
3answers
2k views

what does one mean by numerical integration is too expensive?

I am reading about Bayesian inference and I came across the phrase "numerical integration of the marginal likelihood is too expensive" I do not have a background in mathematics and I was wondering ...
11
votes
1answer
8k views

What is the “capacity” of a machine learning model?

I'm studying this Tutorial on Variational Autoencoders by Carl Doersch. In the second page it states: One of the most popular such frameworks is the Variational Autoencoder [1, 3], the subject of ...
11
votes
1answer
432 views

Variational Inference, KL divergence requires true $p$

To my (very modest) understand of variational inference, one tries to approximate an unknown distribution $p$ by finding a distribution $q$ that optimises the following: $$KL (p||q) = \sum\limits_{x} ...
11
votes
1answer
354 views

What is the difference between VAE and Stochastic Backpropagation for Deep Generative Models?

What is the difference between Auto-encoding Variational Bayes and Stochastic Backpropagation for Deep Generative Models? Does inference in both methods lead to the same results? I'm not aware of any ...
9
votes
2answers
299 views

Uncertainty estimation in high-dimensional inference problems without sampling?

I'm working on a high-dimensional inference problem (around 2000 model parameters) for which we are able to robustly perform MAP estimation by finding the global maximum of the log-posterior using a ...
9
votes
1answer
3k views

Variational autoencoder with Gaussian mixture model

A variational autoencoder (VAE) provides a way of learning the probability distribution $p(x,z)$ relating an input $x$ to its latent representation $z$. In particular, the decoder $d$ maps an input $...
9
votes
3answers
3k views

KL Loss with a unit Gaussian

I've been implementing a VAE and I've noticed two different implementations online of the simplified univariate gaussian KL divergence. The original divergence as per here is $$ KL_{loss}=\log(\frac{\...
9
votes
1answer
218 views

Variational Bayes combined with Monte Carlo

I'm reading up on variational Bayes, and as I understand it, it comes down to the idea that you approximate $p(z\mid x)$ (where $z$ are the latent variables of your model and $x$ the observed data) ...
9
votes
2answers
2k views

Applying stochastic variational inference to Bayesian Mixture of Gaussian

I am trying to implement Gaussian Mixture model with stochastic variational inference, following this paper. This is the pgm of Gaussian Mixture. According to the paper, the full algorithm of ...
8
votes
3answers
4k views

What does a 'tractable' distribution mean?

For example, in generative adversarial network, we often hear that inference is easy because the conditional distribution of x given latent variable z is 'tractable'. Also, I read somewhere that ...
8
votes
1answer
737 views

Variational inference engines

After doing some research on the topic, I have noticed a surprising deficit of inference packages and libraries that rely on message-passing or optimization methods for Python and R. To the best of ...
8
votes
0answers
620 views

Comparing Laplace Approximation and Variational Inference

Does anyone know of any references that look at the relationship between the Laplace approximation and variational inference (with normal approximating distributions)? Namely I'm looking for something ...
7
votes
2answers
654 views

choosing prior parameters for variational mixture of Gaussians

I am implementing a vanilla variational mixture of multivariate Gaussians, as per Chapter 10 of Pattern Recognition and Machine Learning (Bishop, 2007). The Bayesian approach requires to specify (...
7
votes
0answers
196 views

Rao-Blackwellization in variational inference

The Black box VI paper introduces Rao-Blackwellization as a method to reduce the variance of the gradient estimator using score function, in section 3.1. However I don't quite get the basic idea ...
6
votes
1answer
2k views

Deriving the KL divergence loss for VAEs

In a VAE, the encoder learns to output two vectors: $$\mathbf{\mu} \in\ \mathbb{R}^{z}$$ $$\mathbf{\sigma} \in\ \mathbb{R}^{z}$$ which are the mean and variances for the latent vector $\mathbf{z}$, ...
6
votes
1answer
2k views

variational inference with KL

i am self-studying variational inference - and in Murphy's book "A probabilistic perspective on machine learning" it is discussed that minimizing the forward KL divergence (which is stated to be zero-...
6
votes
1answer
134 views

$\min D_\textrm{KL}(p(x_1,\dots,x_n) \mid\mid q_1(x_1)\cdots q_n(x_n))$ gives the marginals of $p(x_1,\dots,x_n)$?

Prove or disprove: Let $p(x_1,\dots,x_n)$ be a given probability distribution over the $n$ variables $x_1, \dots,x_n$. The univariate probability distributions $q(x_1),\dots,q(x_n)$ that minimize the ...
6
votes
0answers
145 views

Is this problem Bayesian? And can I use variational approximation?

Suppose there are $N$ samples of observations $\mathbf X(n)$ ($n=1,\cdots,N$), which are given by probability distribution $p(\mathbf X(n)|\mathbf Z(n))$ with their conditions are given by hidden ...
6
votes
0answers
574 views

Estimation of log-likelihood via importance sampling

I am looking at a model trained with stochastic gradient variational Bayes. In this paper an importance sampler is proposed to estimate the likelihood: $$p(x) \approx {1 \over S} \sum_{s=1}^S {p(x|...
5
votes
2answers
1k views

Why do we use Gaussian distributions in Variational Autoencoder?

I still don't understand why we force the distribution of the hidden representation of a Variational Autoencoder (VAE) to follow a multivariate normal distribution. Why this specific distribution and ...
5
votes
2answers
49 views

What is the relation between “conjugate priors” and the approximate inference?

I know that "conjugate prior" is to help us calculate the the denominator of the Bayes formula(to make the calculations easier). And I just learnt to approximate the inference by mean field ...
5
votes
4answers
5k views

What is the objective of a variational autoencoder (VAE)?

I have read a lot of literature on VAE's and I have understood the basic set-up. However, I still don't know what the overall goal is. The basic set-up is that we have a dataset of observations $\pmb{...
5
votes
3answers
1k views

Variational autoencoder: Why reconstruction term is same to square loss?

In variational autoencoder (see paper), page 5, the loss function for neural networks is defined as: $L(\theta;\phi;x^{i})\backsimeq 0.5*\sum_{j=1}^J(1 + 2\log\sigma^i_j-(\mu^i)^2) - (\sigma^i)^2) + \...
5
votes
1answer
82 views

What is meant by 'Black box variational inference'?

I'm aware of the topic of variational inference (VI) however I'm not really sure what Black box VI is? In particular I am watching a video by David Blei titled Black box variational inference and on ...
5
votes
1answer
698 views

What are the advantages of normalizing flow over VAEs with deep latent gaussian models for inference?

I am reading the normalizing flow paper and am a bit confused. It seems that being able to model complex (correlated?) posterior is one of the advantages of the proposed approach (Section 2.3, last ...
5
votes
1answer
82 views

how does one compute expectations for non-linear functions

I am continuing my struggles with approximate Bayesian inference methods. I have a fundamental doubt about how to compute certain expectations that arise during variational bayes, for example. So, my ...
5
votes
1answer
219 views

Help with Variational Bayes on a weighted linear regression model

I am trying to setup VB to do a weighted linear regression for vector observations. My setup is that I have $N$ numbers of $d$-dimensional vector observations. I would like to model the noise as being ...
5
votes
0answers
269 views

free energy of variational autoencoder

There is a conception of free energy related to restricted boltzmann machine(RBM). Since variational autoencoder(VAE) is an alternative to RBM for autoencoding, is there a counterpart definition of ...
4
votes
2answers
4k views

Loss function autoencoder vs variational-autoencoder or MSE-loss vs binary-cross-entropy-loss

When having real valued entries (e.g. floats between 0 and 1 as normalized representation for greyscale values from 0 to 256) in our label vector, I always thought that we use MSE(R2-loss) if we want ...
4
votes
2answers
1k views

Reparameterization trick for gamma distribution

I am reading the work of Welling on Vartiational Auto-Encoders (VAE), and wonder if there is any way to generate Gamma distributed samples via a similar reparametrization? The idea of ...
4
votes
2answers
670 views

Can a posterior expectation be used as a approximate for the true (prior) expectation?

Let's say that the likelihood of observation $x$ given a random latent variable $z$ and a model parameter $\theta$ is defined as $p(x|\theta, z)$. As far as I know, if I want to obtain $p(x| \theta)$,...
4
votes
1answer
881 views

Variational Bayes: Understanding Mean field approximation

I am looking at the mean field approximation as used in Variational Bayes inference and I looked at this section on wikipedia with the factorised approximation as described here: https://en.wikipedia....
4
votes
1answer
566 views

When generating samples using variational autoencoder, we decode samples from $N(0,1)$ instead of $\mu + \sigma N(0,1)$

Context: I'm trying to understand the use of variational autoencoders as generators. My understanding: During training, for an input point $x_i$ we want to learn latent $\mu_i$ and $\sigma_i$ and ...
4
votes
1answer
628 views

How to obtain the functional derivative in variational inference?

Referring to David Blei's notes on variational inference, I wonder how to get the derivative of $q(z_k)$, the distribution of $z_k$, in eq. 23 from eq. 22. Namely, eq. 22 is $L_k = \int q(z_k) \...
4
votes
1answer
452 views

Why do we use the mean-field approximation for variational Bayes?

I often see the mean-field approximation for Variational Bayes. I understand the independence assumption: what I don't understand is why we make that assumption. How does it help us?
4
votes
1answer
212 views

Why is random sampling a non-differentiable operation?

This answer states that we cannot back-propagate through a random node. So, in the case of VAEs, you have the reparametrisation trick, which shifts the source of randomness to another variable ...
4
votes
1answer
162 views

Why do Variational Bayes methods assume that the likelihood $p(x|z)$ is tractable while the posterior is not?

I am trying to understand the motivation behind Variational Bayes. I get that the posterior $p(z|x)$ can be intractable, when we would have to compute the evidence with $p(x) = \int p(x|z)p(z) \text{d}...
4
votes
1answer
427 views

What are the downsides of bayesian neural networks?

Bayesian neural nets (BNN) are very popular topic. With development of variational approximation it became possible to train such models much faster then with Monte Carlo sampling. BNNs allow such ...
4
votes
2answers
177 views

A slick expectation calculation: how was it done?

$\newcommand{\Cov}{\mathrm{Cov}}$ We have that $$ \Delta_{\lambda} KL(\lambda) = \mathbb{E}_{\theta\sim q_{\lambda}(\theta),z\sim g_{N}(z|\theta)}(\Delta_{\lambda}[\log \: q_{\lambda}(\theta)](\log \...
4
votes
1answer
234 views

What is the idea behind Bayes By Backprop?

Having looked through the internet and the paper, I find Bayes by Backprop very unaccesible for my intermediate understanding of variational inference. Most online guides also lack some explaining ...
4
votes
1answer
257 views

Variational Inference

I have been looking at variational inference, however I came across a problem I have a hard time to understand how to start with it. Let $$\mu_n \sim N(\mu', \sigma'^2) \\ \tau_k \sim N(\tau', \...
4
votes
0answers
405 views

Why maximizing the lower bound of variational evidence maximizes the probability of observing data

In vaiational bayesian inference we attempt to find a proxy function to best estimate the intractable posterior $P(z|X)$. We define best as the probability distribution that minimizes the KL ...
4
votes
1answer
59 views

In variational inference on von Mises clusters, how to find a bound for the log-Bessel function?

This paper on von Mises clustering uses an upper bound on the modified log-Bessel function that I struggle to replicate. Taking results from this paper, the authors state: $$u\frac{I'_\nu(u)}{I_\nu(u)...
4
votes
0answers
376 views

How can variational inference (for LDA) be explained in layman's terms?

I am learning probabilistic topic modeling, and I am studying latent Dirichlet allocation, specifically the inference process. I found many mathematical details, but I need a layman's explanation and/...