Questions tagged [variational-bayes]

Variational Bayesian methods approximate intractable integrals found in Bayesian inference and machine learning. Primarily, these methods serve one of two purposes: Approximating the posterior distribution, or bounding the marginal likelihood of observed data.

Filter by
Sorted by
Tagged with
1
vote
0answers
225 views

How do we train variational autoencoders in practice?

I learnt that the objective function of a VAE is given by the RHS of the equation $$\ln p(x_n)-KL(q(z|x_n)\Vert p(z|x_n)) = \Bbb E_{z \sim q(z|x_n)}(\ln p(x_n|z))-KL(q(z|x_n) \Vert p(z))$$ in which $...
2
votes
1answer
803 views

Sampling z in VAE

How many times do we sample from $Q(z|x)$ in a Variational Autoencoder? Let’s say that the autoencoder input $x$ is a single image 28x28 pixels - and $Z$ is is a one dimensional distribution. Then, ...
1
vote
1answer
205 views

Variational autoencoder obtain high likelihood but produce low quality sample?

I'm watching Ian Goodfellow's introduction to generative models. When he was introducing variational autoencoders at 22:29, he said: Variational autoencoders are good at obtaining high likelihood,...
2
votes
0answers
55 views

Hierarchical Black Box Variational Inference : Choice of inverse flow

I am reading through Black Box Variational Inference, and having trouble understanding the section for hierarchical inference, where the normalizing flow is introduced. Should this be an arbitrary ...
3
votes
1answer
415 views

Diversity of generated samples in VAE

In a variational autoencoder (VAE), it is possible to generate new samples (i.e. images) based on the latent space. After having read quite a few papers about VAE, I still wonder what drives the ...
1
vote
0answers
54 views

How to interpret graphical model for Dirichlet process mixture for variational inference?

I am working through this paper by Blei and Jordan, which introduces variational inference for Dirichlet process mixtures. They derive an evidence lower bound (ELBO) function based on a stick breaking ...
1
vote
0answers
19 views

Why assumptions of exponential family on the complete conditionals imply a conjugacy relationship on other variables

I am reading the Stochastic Variational Inference paper. Basically it first assumes that the joint distribution factorizes into a global term and a product of local terms. $$p(x, z, \beta | \alpha) = ...
2
votes
1answer
557 views

Why in Variational Auto Encoder (Gaussian variational family) we model $\log\sigma^2$ and not $\sigma^2$ (or $\sigma$) itself?

In theory the encoder in VAE (assuming that variational family is Gaussian) generates the $\mu$ and $\sigma$ (or $\sigma^2$). But, in practice, I have seen people assuming the output is $\log\sigma^2$....
10
votes
1answer
339 views

What is the difference between VAE and Stochastic Backpropagation for Deep Generative Models?

What is the difference between Auto-encoding Variational Bayes and Stochastic Backpropagation for Deep Generative Models? Does inference in both methods lead to the same results? I'm not aware of any ...
4
votes
1answer
145 views

Why do Variational Bayes methods assume that the likelihood $p(x|z)$ is tractable while the posterior is not?

I am trying to understand the motivation behind Variational Bayes. I get that the posterior $p(z|x)$ can be intractable, when we would have to compute the evidence with $p(x) = \int p(x|z)p(z) \text{d}...
8
votes
1answer
2k views

Variational autoencoder with Gaussian mixture model

A variational autoencoder (VAE) provides a way of learning the probability distribution $p(x,z)$ relating an input $x$ to its latent representation $z$. In particular, the decoder $d$ maps an input $...
4
votes
2answers
3k views

Loss function autoencoder vs variational-autoencoder or MSE-loss vs binary-cross-entropy-loss

When having real valued entries (e.g. floats between 0 and 1 as normalized representation for greyscale values from 0 to 256) in our label vector, I always thought that we use MSE(R2-loss) if we want ...
3
votes
0answers
94 views

What are the current popular methods for Large Scale Gaussian Process Regression, and which of them are readily available in R?

Vanilla Gaussian Process Regression requires $O(N^3)$ multiplications for estimation, $O(N^2)$ multiplications for prediction and it uses $O(N^2)$ memory where $N$ is the sample size, so it's not ...
5
votes
3answers
1k views

Variational autoencoder: Why reconstruction term is same to square loss?

In variational autoencoder (see paper), page 5, the loss function for neural networks is defined as: $L(\theta;\phi;x^{i})\backsimeq 0.5*\sum_{j=1}^J(1 + 2\log\sigma^i_j-(\mu^i)^2) - (\sigma^i)^2) + \...
3
votes
1answer
136 views

What can we learn from this visualization?

In here, I learned that the learned MNIST manifold can be visualized as the image below (on page 10, figure 4(b)). My understanding for this visualization is: We start from probability integral ...
1
vote
1answer
80 views

What is a $\propto$ update in mean-field approximation?

I am trying to understand some notation used in papers about Bayesian variational inference. In some papers that use mean-field approximation to fit a probabilistic model, they describe coordinate ...
0
votes
1answer
283 views

VAE latent vector not taking unit normal distribution

I trained a Convolutional VAE on 1-D electric load curve data (one sample consists of 48-time steps). The training loss (mean square error + KL divergence) decreases during training and is converging. ...
1
vote
1answer
63 views

VAEs: Basic Math example for pass through VAE

As great as many papers and videos are at teaching Neural Net concepts, I see a surprising lack of basic numerical examples explaining these concepts. These examples would let me make sure I know ...
1
vote
0answers
81 views

Jensen's inequality in Collaborative Topic Regression

I am reading the article Collaborative Topic Modeling for Recommending Scientific Articles and could notice the application of Jensen's inequality in order to define a bound from which optimization is ...
0
votes
1answer
382 views

What is the loss function for a probabilistic decoder in the Variational Autoencoder?

In a VAE with Gaussian output the loss function is usually:$$\sum{(\hat x - x)^2} + KL,$$ so the sum of squared errors plus KL divergence. When I also want to predict the variance of the reconstructed ...
2
votes
2answers
256 views

How well does $Q(z|X)$ match $N(0,I)$ in variational autoencoders?

At train time, the KL divergence term drives $Q(z=\mu(X)+\epsilon \times\Sigma(X) | X)$ toward $N(0,I)$, where $\epsilon\sim N(0,I)$. It can't drive $Q(z|X)$ to exactly $N(0,I)$ because the ...
24
votes
2answers
8k views

how to weight KLD loss vs reconstruction loss in variational auto-encoder

in nearly all code examples I've seen of a VAE, the loss functions are defined as follows (this is tensorflow code, but I've seen similar for theano, torch etc. It's also for a convnet, but that's ...
0
votes
2answers
465 views

latent dirichlet allocation: complexity and implementation details

I was confused by how LDA (by the original variational inference) can be implemented in a way such that the number of operations for each document $j$ is $\mathcal{O}(N_j~K)$, where $N_j$ is the ...
2
votes
1answer
178 views

Variational Inference of Univariate Gaussian mixtures

I am reading this paper. In the paper, they use an example of mixture of unit-variance univariate Gaussians with the following parameterization: \begin{align} \mu_k & \sim \mathcal{N}(0, \sigma^2)...
1
vote
0answers
340 views

Comparing ELBO of a VAE for different samples

I am lacking of an interpretation of the evidence lower bound (ELBO), when comparing two different samples $x_1, x_2 \sim X$. Writing the marginal log-likelihood as the sum of lower variational bound ...
3
votes
1answer
287 views

Derivation of Variational Inference

I'm reading Blei et al. (2017) "Variational Inference: A Review for Statisticians" to understand Variational Inference (VI). I follow the paper's notations: $\mathbf{x}_{1:n}$ (observations), $\mathbf{...
1
vote
2answers
720 views

Variational inference: how to rewrite ELBO?

I am reading this paper on variational inference and this website. One thing I am confused about is how they get to decompose ELBO, where $ELBO(q) = E_q[log~p(z,x)] - E_q[log~q(z)]$, when focusing ...
2
votes
1answer
341 views

Computing the gradient of the expectation of a function w.r.t the parameters of the distribution

I am reading this paper on variational auto-encoders by Kingma & Welling and in section 2.2 authors write the following equality for the gradient of the expectation of a function with respect to ...
1
vote
1answer
849 views

Difference between stochastic variational inference and variational inference?

Very simple, as the question header says: what is the difference between SVI and VI? I cannot seem to find a clear-cut definition online.
1
vote
0answers
105 views

Questions about Mean-field variational inference

I am very new to this variational inference concept. I couldn't find any clear sources. I have two questions related to each other. Let's consider a very simple probabilistic model with a 2-D latent ...
1
vote
0answers
489 views

How to write loss function for variational autoencoder?

So I've trying to follow various resources (Geron, Doersch, Altesaar, et al.) to construct a working loss function for my variational autoencoder but I'm finding that formulations either seem to work ...
17
votes
4answers
6k views

When should I use a variational autoencoder as opposed to an autoencoder?

I understand the basic structure of variational autoencoder and normal (deterministic) autoencoder and the math behind them, but when and why would I prefer one type of autoencoder to the other? All I ...
0
votes
1answer
89 views

High Dimensional Visualization and VAE Validation

I have a dataset from a black box function, about 35K lines in a text file, with each line containing a single string from the black box function. I am building a VAE to (hopefully) model that data, ...
24
votes
1answer
4k views

What are variational autoencoders and to what learning tasks are they used?

As per this and this answer, autoencoders seem to be a technique that uses neural networks for dimension reduction. I would like to additionally know what is a variational autoencoder (its main ...
3
votes
1answer
89 views

The explanation for the need to compute (rather then optimize) the posterior of latent variables

The most common usage of the variational inference looks like to be in computing the marginal distribution $P(X)$ in the denominator of the Bayes formula when computing the posterior probability of ...
3
votes
1answer
165 views

Is value of ELBO a scalar or a distribution?

I'm trying to get my head around variational inference, but I'm confused with the definition of ELBO, specifically an expectation over joint distribution. Here I use Variational Inference: A Review ...
8
votes
0answers
559 views

Comparing Laplace Approximation and Variational Inference

Does anyone know of any references that look at the relationship between the Laplace approximation and variational inference (with normal approximating distributions)? Namely I'm looking for something ...
2
votes
1answer
75 views

In VAEs, why don't we just use a fixed variance for the z distribution?

I'm practicing with VAEs for generative purpose, and from what I understood we need the latent variable $z$ to approximate a distribution, usually the standard normal $N(0, I)$. In any example I ...
5
votes
1answer
2k views

Deriving the KL divergence loss for VAEs

In a VAE, the encoder learns to output two vectors: $$\mathbf{\mu} \in\ \mathbb{R}^{z}$$ $$\mathbf{\sigma} \in\ \mathbb{R}^{z}$$ which are the mean and variances for the latent vector $\mathbf{z}$, ...
9
votes
2answers
3k views

KL Loss with a unit Gaussian

I've been implementing a VAE and I've noticed two different implementations online of the simplified univariate gaussian KL divergence. The original divergence as per here is $$ KL_{loss}=\log(\frac{\...
3
votes
1answer
330 views

Help with derivation of Mean Field Variational Inference

I am studying Variational Inference using Bishop's book: Pattern Recognition and Machine Learning. At the moment, I am struggling to understand the Lower Bound derivation for the Mean-Field ...
1
vote
1answer
101 views

Bethe approximation for factor graphs

I am confused at computing Bethe approximation for factor graphs in here. It generalizes Bethe approxmiation in a pairwise case. However, I am wondering why (75) goes to (78) with (76): We can verify ...
4
votes
0answers
373 views

Why maximizing the lower bound of variational evidence maximizes the probability of observing data

In vaiational bayesian inference we attempt to find a proxy function to best estimate the intractable posterior $P(z|X)$. We define best as the probability distribution that minimizes the KL ...
0
votes
1answer
119 views

How to compute the Gibbs free energy in Bethe approximation for MRF

Hi, I am learning loopy belief propagation for MRF. The general roadmap is to define a Bethe approximation, which is exact for a tree but wrong for general graphs. I'm currently stuck at the point to ...
2
votes
0answers
101 views

What is the result for EM algorithm in smoothed LDA model?

In the original LDA paper (Blei2003), EM algorithm estimates $\alpha$ and $\beta$ in Fig.5. So, what is the result for Fig.7? Will it give estimation of $\alpha, \beta$ or $\alpha, \eta$? And, if I ...
3
votes
0answers
61 views

Simple approximation of joint posterior

Consider the (hierarchical) Bayesian inference problem with two unknowns $(x,\theta)$ and data $y$. I'm using a very simple ("independence"?) approximation $$ p(x,\theta|y) \approx p(x|\theta_\star,y) ...
2
votes
2answers
927 views

If z is uniform normal is $f(z;\theta)$ a normal distribution

I am trying to understand the description of variational auto-encoders here to quote the excerpt: Before we can say that our model is representative of our dataset, we need to make sure that for ...
3
votes
1answer
168 views

Why is p(x|z) tractable but p(z|x) intractable?

In variational methods, given a set of latent variables $z$ corresponding to visible variables $x$, why is it that the probability distribution $p\left(x\middle|z\right)$ is tractable to compute, but $...
10
votes
1answer
8k views

What is the “capacity” of a machine learning model?

I'm studying this Tutorial on Variational Autoencoders by Carl Doersch. In the second page it states: One of the most popular such frameworks is the Variational Autoencoder [1, 3], the subject of ...
1
vote
1answer
280 views

Proving that the Variational EM algorithm converges

We know that \begin{align} \log p(x) &=\mathbb{E}_{z\sim q(z|x)}\left[\log p(x)\right] \\ &=\mathbb{E}_{z\sim q(z|x)}\left[\log\left(p(x)\frac{p(z|x)q(z|x)}{p(z|x)q(z|x)}\right)\right] \\...