Questions tagged [variational-bayes]

Variational Bayesian methods approximate intractable integrals found in Bayesian inference and machine learning. Primarily, these methods serve one of two purposes: Approximating the posterior distribution, or bounding the marginal likelihood of observed data.

Filter by
Sorted by
Tagged with
120 votes
9 answers
97k views

How does the reparameterization trick for VAEs work and why is it important?

How does the reparameterization trick for variational autoencoders (VAE) work? Is there an intuitive and easy explanation without simplifying the underlying math? And why do we need the 'trick'?
David Dao's user avatar
  • 2,824
56 votes
5 answers
41k views

How to weight KLD loss vs reconstruction loss in variational auto-encoder?

in nearly all code examples I've seen of a VAE, the loss functions are defined as follows (this is tensorflow code, but I've seen similar for theano, torch etc. It's also for a convnet, but that's ...
memo's user avatar
  • 999
52 votes
1 answer
16k views

Variational inference versus MCMC: when to choose one over the other?

I think I get the general idea of both VI and MCMC including the various flavors of MCMC like Gibbs sampling, Metropolis Hastings etc. This paper provides a wonderful exposition of both methods. I ...
kedarps's user avatar
  • 3,532
51 votes
4 answers
34k views

When should I use a variational autoencoder as opposed to an autoencoder?

I understand the basic structure of variational autoencoder and normal (deterministic) autoencoder and the math behind them, but when and why would I prefer one type of autoencoder to the other? All I ...
DiveIntoML's user avatar
  • 2,013
40 votes
3 answers
29k views

What is the "capacity" of a machine learning model?

I'm studying this Tutorial on Variational Autoencoders by Carl Doersch. In the second page it states: One of the most popular such frameworks is the Variational Autoencoder [1, 3], the subject of ...
Andrés Marafioti's user avatar
36 votes
2 answers
9k views

What are variational autoencoders and to what learning tasks are they used?

As per this and this answer, autoencoders seem to be a technique that uses neural networks for dimension reduction. I would like to additionally know what is a variational autoencoder (its main ...
user avatar
36 votes
1 answer
11k views

Relation between variational Bayes and EM

I read somewhere that Variational Bayes method is a generalization of the EM algorithm. Indeed, the iterative parts of the algorithms are very similar. In order to test whether the EM algorithm is a ...
Ufuk Can Bicici's user avatar
29 votes
1 answer
20k views

Deriving the KL divergence loss for VAEs

In a VAE, the encoder learns to output two vectors: $$\mathbf{\mu} \in\ \mathbb{R}^{z}$$ $$\mathbf{\sigma} \in\ \mathbb{R}^{z}$$ which are the mean and variances for the latent vector $\mathbf{z}$, ...
YellowPillow's user avatar
  • 1,241
26 votes
3 answers
25k views

Loss function autoencoder vs variational-autoencoder or MSE-loss vs binary-cross-entropy-loss

When having real valued entries (e.g. floats between 0 and 1 as normalized representation for greyscale values from 0 to 256) in our label vector, I always thought that we use MSE(R2-loss) if we want ...
SolingerStuebchen's user avatar
23 votes
3 answers
11k views

What does a 'tractable' distribution mean?

For example, in generative adversarial network, we often hear that inference is easy because the conditional distribution of x given latent variable z is 'tractable'. Also, I read somewhere that ...
sirius27's user avatar
  • 341
20 votes
2 answers
13k views

Why do we use Gaussian distributions in Variational Autoencoder?

I still don't understand why we force the distribution of the hidden representation of a Variational Autoencoder (VAE) to follow a multivariate normal distribution. Why this specific distribution and ...
Tbertin's user avatar
  • 399
20 votes
1 answer
5k views

Why in Variational Auto Encoder (Gaussian variational family) we model $\log\sigma^2$ and not $\sigma^2$ (or $\sigma$) itself?

In theory the encoder in VAE (assuming that variational family is Gaussian) generates the $\mu$ and $\sigma$ (or $\sigma^2$). But, in practice, I have seen people assuming the output is $\log\sigma^2$....
user3639557's user avatar
  • 1,442
19 votes
1 answer
28k views

How should I intuitively understand the KL divergence loss in variational autoencoders? [duplicate]

I was studying VAEs and came across the loss function that consists of the KL divergence. $$ \sum_{i=1}^n \sigma^2_i + \mu_i^2 - \log(\sigma_i) - 1 $$ I wanted to intuitively make sense of the KL ...
raptorAcrylyc's user avatar
16 votes
2 answers
3k views

Comparing Laplace Approximation and Variational Inference

Does anyone know of any references that look at the relationship between the Laplace approximation and variational inference (with normal approximating distributions)? Namely I'm looking for something ...
aleshing's user avatar
  • 1,568
14 votes
1 answer
9k views

Is the output of a variational autoencoder meant to be a distribution that can be sampled, or a sample directly?

It is difficult to ask this question succinctly in the title, so let me explain. From all the examples of VAEs I have seen, there seem to be 2 approaches used to implement them. In these, ...
Chechy Levas's user avatar
  • 1,245
14 votes
3 answers
2k views

Is the optimization of the Gaussian VAE well-posed?

In a Variational Autoencoder (VAE), given some data $x$ and latent variables $t$ with prior distribution $p(t) = \mathcal{N}(t \mid 0, I)$, the encoder aims to learn a distribution $q_{\phi}(t)$ that ...
learner's user avatar
  • 697
14 votes
1 answer
679 views

Variational Inference, KL divergence requires true $p$

To my (very modest) understand of variational inference, one tries to approximate an unknown distribution $p$ by finding a distribution $q$ that optimises the following: $$KL (p||q) = \sum\limits_{x} ...
Vincent Warmerdam's user avatar
13 votes
3 answers
3k views

what does one mean by numerical integration is too expensive?

I am reading about Bayesian inference and I came across the phrase "numerical integration of the marginal likelihood is too expensive" I do not have a background in mathematics and I was wondering ...
discretetimeisnice's user avatar
13 votes
3 answers
13k views

Variational autoencoder: Why reconstruction term is same to square loss?

In variational autoencoder (see paper), page 5, the loss function for neural networks is defined as: $L(\theta;\phi;x^{i})\backsimeq 0.5*\sum_{j=1}^J(1 + 2\log\sigma^i_j-(\mu^i)^2) - (\sigma^i)^2) + \...
null's user avatar
  • 405
13 votes
2 answers
8k views

KL Loss with a unit Gaussian

I've been implementing a VAE and I've noticed two different implementations online of the simplified univariate gaussian KL divergence. The original divergence as per here is $$ KL_{loss}=\log(\frac{\...
groovyDragon's user avatar
12 votes
1 answer
11k views

Variational autoencoder with Gaussian mixture model

A variational autoencoder (VAE) provides a way of learning the probability distribution $p(x,z)$ relating an input $x$ to its latent representation $z$. In particular, the encoder $e$ maps an input $...
D.W.'s user avatar
  • 6,668
11 votes
2 answers
3k views

What is the relationship between VAE and EM algorithm?

I know that the EM algorithm is used in latent variable models, specifically to do maximum likelihood estimation iteratively. Similarly, the VAE can be used for latent variable models and, although ...
MLQuestioner's user avatar
11 votes
1 answer
2k views

Why variational inference and not maximum likelihood?

When using variational inference scheme, we assume latent variable $\mathbf z$, model $p(\mathbf x, \mathbf z)$, and maximize $\log p(\mathbf x)$. Introducing variational distribution $q(\mathbf z)$, ...
user's user avatar
  • 111
11 votes
1 answer
9k views

Variational Auto Encoder (VAE) sampling from prior vs posterior

I have been reading the original VAE paper,Auto-Encoding Variational Bayes. In VAE, when generating samples, why do we sample from prior instead of the learned variational posterior(Fig 5 in the paper)...
p__10's user avatar
  • 181
11 votes
2 answers
515 views

Uncertainty estimation in high-dimensional inference problems without sampling?

I'm working on a high-dimensional inference problem (around 2000 model parameters) for which we are able to robustly perform MAP estimation by finding the global maximum of the log-posterior using a ...
CBowman's user avatar
  • 613
11 votes
1 answer
270 views

Variational Bayes combined with Monte Carlo

I'm reading up on variational Bayes, and as I understand it, it comes down to the idea that you approximate $p(z\mid x)$ (where $z$ are the latent variables of your model and $x$ the observed data) ...
Peter Bloem's user avatar
  • 1,119
10 votes
1 answer
3k views

When generating samples using variational autoencoder, we decode samples from $N(0,1)$ instead of $\mu + \sigma N(0,1)$

Context: I'm trying to understand the use of variational autoencoders as generators. My understanding: During training, for an input point $x_i$ we want to learn latent $\mu_i$ and $\sigma_i$ and ...
Edward B.'s user avatar
  • 103
10 votes
1 answer
5k views

Sampling z in VAE

How many times do we sample from $Q(z|x)$ in a Variational Autoencoder? Let’s say that the autoencoder input $x$ is a single image 28x28 pixels - and $Z$ is is a one dimensional distribution. Then, ...
Edv Beq's user avatar
  • 703
10 votes
3 answers
3k views

Applying stochastic variational inference to Bayesian Mixture of Gaussian

I am trying to implement Gaussian Mixture model with stochastic variational inference, following this paper. This is the pgm of Gaussian Mixture. According to the paper, the full algorithm of ...
user5779223's user avatar
10 votes
1 answer
806 views

What is the difference between VAE and Stochastic Backpropagation for Deep Generative Models?

What is the difference between Auto-encoding Variational Bayes and Stochastic Backpropagation for Deep Generative Models? Does inference in both methods lead to the same results? I'm not aware of any ...
Dion's user avatar
  • 944
9 votes
3 answers
2k views

What is meant by 'Black box variational inference'?

I'm aware of the topic of variational inference (VI) however I'm not really sure what Black box VI is? In particular I am watching a video by David Blei titled Black box variational inference and on ...
piccolo's user avatar
  • 937
9 votes
2 answers
5k views

Gradient of the expectation of a function w.r.t. distribution parameters

In section 2.2 of Kingma & Welling's paper on variational auto-encoders authors write the following equality for the gradient of the expectation of a function with respect to the parameters of the ...
ted's user avatar
  • 641
9 votes
2 answers
3k views

Why is random sampling a non-differentiable operation?

This answer states that we cannot back-propagate through a random node. So, in the case of VAEs, you have the reparametrisation trick, which shifts the source of randomness to another variable ...
user avatar
9 votes
2 answers
2k views

VAE: why we do not sample again after decoding and before reconstruction loss?

In many of the VAE schematics and in the original paper, a sampling step is present after decoding and before the reconstruction loss as shown in the image below. The image comes from Stanford CS321n. ...
Luca Angioloni's user avatar
9 votes
1 answer
3k views

variational inference with KL

i am self-studying variational inference - and in Murphy's book "A probabilistic perspective on machine learning" it is discussed that minimizing the forward KL divergence (which is stated to be zero-...
Wouter's user avatar
  • 2,182
9 votes
2 answers
852 views

Variational inference engines

After doing some research on the topic, I have noticed a surprising deficit of inference packages and libraries that rely on message-passing or optimization methods for Python and R. To the best of ...
Amelio Vazquez-Reina's user avatar
9 votes
2 answers
2k views

choosing prior parameters for variational mixture of Gaussians

I am implementing a vanilla variational mixture of multivariate Gaussians, as per Chapter 10 of Pattern Recognition and Machine Learning (Bishop, 2007). The Bayesian approach requires to specify (...
lacerbi's user avatar
  • 5,186
9 votes
1 answer
2k views

Why maximizing the lower bound of variational evidence maximizes the probability of observing data

In vaiational bayesian inference we attempt to find a proxy function to best estimate the intractable posterior $P(z|X)$. We define best as the probability distribution that minimizes the KL ...
figs_and_nuts's user avatar
9 votes
1 answer
493 views

Rao-Blackwellization in variational inference

The Black box VI paper introduces Rao-Blackwellization as a method to reduce the variance of the gradient estimator using score function, in section 3.1. However I don't quite get the basic idea ...
avocado's user avatar
  • 3,571
8 votes
2 answers
5k views

Reparameterization trick for gamma distribution

I am reading the work of Welling on Vartiational Auto-Encoders (VAE), and wonder if there is any way to generate Gamma distributed samples via a similar reparametrization? The idea of ...
user3639557's user avatar
  • 1,442
8 votes
4 answers
9k views

What is the objective of a variational autoencoder (VAE)?

I have read a lot of literature on VAE's and I have understood the basic set-up. However, I still don't know what the overall goal is. The basic set-up is that we have a dataset of observations $\pmb{...
Lemon's user avatar
  • 289
8 votes
4 answers
1k views

How well does $Q(z|X)$ match $N(0,I)$ in variational autoencoders?

At train time, the KL divergence term drives $Q(z=\mu(X)+\epsilon \times\Sigma(X) | X)$ toward $N(0,I)$, where $\epsilon\sim N(0,I)$. It can't drive $Q(z|X)$ to exactly $N(0,I)$ because the ...
foghorn's user avatar
  • 233
8 votes
1 answer
2k views

Why aren't auto-encoders also considered generative models?

Auto-encoders (AEs) are composed of an encoder and a decoder (often represented by a neural network). The encoder produces a vector representation $z$ of its input $x$ (e.g. an image). The decoder ...
user avatar
8 votes
2 answers
3k views

What are the downsides of bayesian neural networks?

Bayesian neural nets (BNN) are very popular topic. With development of variational approximation it became possible to train such models much faster then with Monte Carlo sampling. BNNs allow such ...
Leonid's user avatar
  • 255
8 votes
2 answers
4k views

Should reconstruction loss be computed as sum or average over input for variational autoencoders?

I am following this variational autoencoder tutorial: https://keras.io/examples/generative/vae/. I have included the loss computation part of the code below. I know VAE's loss function consists of the ...
Jane Sully's user avatar
  • 1,030
8 votes
1 answer
362 views

Mathematical notation for suppressing differentiation

Basic question Is the some existing mathematical notation to mean "treat this term as a constant when differentiating"? This would be the equivalent of ...
Dennis Prangle's user avatar
8 votes
1 answer
2k views

What is the idea behind Bayes By Backprop?

Having looked through the internet and the paper, I find Bayes by Backprop very inaccessible for my intermediate understanding of variational inference. Most online guides also lack some explaining ...
boomkin's user avatar
  • 855
8 votes
0 answers
1k views

Estimation of log-likelihood via importance sampling

I am looking at a model trained with stochastic gradient variational Bayes. In this paper an importance sampler is proposed to estimate the likelihood: $$p(x) \approx {1 \over S} \sum_{s=1}^S {p(x|...
bayerj's user avatar
  • 13.7k
7 votes
1 answer
1k views

reparameterization trick in VAEs, How should we do this?

I'm confused about how does reparameterization trick works. In this article shows it very simple. You learn two vectors $\sigma$ and $\mu$, sample $\epsilon$ from $N(0, 1)$ and then your latent ...
Peyman's user avatar
  • 309
7 votes
1 answer
616 views

Why do Variational Bayes methods assume that the likelihood $p(x|z)$ is tractable while the posterior is not?

I am trying to understand the motivation behind Variational Bayes. I get that the posterior $p(z|x)$ can be intractable, when we would have to compute the evidence with $p(x) = \int p(x|z)p(z) \text{d}...
Kilian Obermeier's user avatar

1
2 3 4 5
8