Questions tagged [variational-bayes]

Variational Bayesian methods approximate intractable integrals found in Bayesian inference and machine learning. Primarily, these methods serve one of two purposes: Approximating the posterior distribution, or bounding the marginal likelihood of observed data.

Filter by
Sorted by
Tagged with
0
votes
1answer
12 views

Reparametrization trick in VAE at Evaluation time

So I've been trying to implement the Variational Auto-Encoder model of Kigma et.al, but something has been bugging me. While I understand the need for reparametrization trick at training time, the ...
0
votes
0answers
8 views

Computing the variational distribution of a softmax function

I am trying to compute the variational approximate distribution of the following softmax function \begin{equation} \mathbb{E}_{q(\mathbf{s}_{t-1})q(\mathbf{x}_t)}\big[P(\mathbf{s}_{t}=i|\mathbf{s}...
0
votes
0answers
32 views

Why using variational inference to do minimization? [closed]

I'm reading a paper on conditoinal random fields. They arrive at a formulation for energy, and they go like this: "minimizing this is intractable" What does that mean? I heard about intractable ...
0
votes
1answer
26 views

Why do we penalize individual example divergence in variational autoencoder?

In variational autoencoder, we want to learn a mapping between input space $X$ and latent space $Z$, and $z\in Z$ is related to $x\in X$ with $z\sim MVN(\mu(x), \Sigma(x))$. In addition, we desire ...
1
vote
0answers
18 views

Need some help understanding the factorised posterior in semi-supervised generative modelling

I am having a bit of trouble with the derivation in Kingma's semi-supervised generative modelling paper for the M2-model. The M2 model assumes a probabilistic model where the data $x$ is generated by ...
0
votes
0answers
44 views

VRNN (Variational Recurrent Neural Network) code with Variable Input Lengths on Tensorflow

I have been trying to write VRNN (Variational Recurrent Neural Network: A recurrent latent variable model for sequential data (NIPS 2015)) with variable input data length. My problem is that it is ...
0
votes
1answer
47 views

What is the difference between approximate bayesian computation vs approximate bayesian inference?

What are the main differences between approximate bayesian computation vs approximate bayesian inference? Are they essentially the same? Do they refer to the same of different family of models? My ...
0
votes
0answers
15 views

Evaluating General Variational Inference Local and Global Updates

For coordinate ascent variational inference (CAVI) we can iterate between the "local" parameter updates and the "global" parameter updates. These can be expressed (following [A]) for exponential ...
0
votes
0answers
22 views

Conditional mutual information of latent variables in a VAE

Consider a VAE with two latent variables, continuous $z$ and discrete $y$. The variational inference distribution $q(z,y|x)$ does not assume mean field factorization, so we compute it as $q(z|x,y)q(y|...
1
vote
1answer
156 views

Gradients of KL divergence and ELBO for variational inference

When doing variational inference, due to intractability we typically maximize the evidence lower bound (ELBO) instead of minimizing Kullback-Leibler divergence (KLD) between our approximate and exact ...
1
vote
2answers
46 views

Semi-supervised objective function VAE

In Kingma's paper on Semi-supervised learning https://arxiv.org/pdf/1406.5298.pdf, we are shown equations for the ELBO for the semisupervised case, however I am having a hard trying to derive the math ...
0
votes
0answers
38 views

Is “variational approximation” synonymous with “variational inference”?

The title says it all. I am currently reading up on deep generative models, and frequently encounter the term "variational inference" as well as the term "variational approximation" to refer to what ...
0
votes
0answers
22 views

Shouldn't the distribution of alpha be included in the predictive distribution of variational linear regression?

I was reading "pattern recognition and machine learning" book and I see that in the predictive distribution of the bayesian linear regression with variatioanl inference, it uses this equation $p(t|x, ...
0
votes
2answers
215 views

Obtaining VAE reconstruction probability

How does one calculate the reconstruction probability? Let's look at the keras example code from here. Is the reconstruction probability the output of a specific layer, or is it to be calculated ...
1
vote
1answer
28 views

Shouldn't we sample from the output of variational auto-encoder?

I know that the output of the VAE is the parameters of the data. For example: If the data follows normal distribution $X \sim \mathcal{N}(\mu,\sigma)$, the generative network should output $\mu$ and $\...
0
votes
1answer
33 views

Does variational auto-encoder output the variational distribution of the latent variable or the distribution of the input x?

In the simple case of mixture of gaussians(with known variance), we have 2 latent variables $\mu$ and $z$. In the vaiational auto-encoder, we assume that the model is infinite mixture of gaussians. If ...
1
vote
1answer
30 views

Why does variational bayes use $KL(Q || P)$ and not $KL(P || Q)$

In variational Bayes, we approximate the intractable posterior $P(Z | X)$ with a tractable $Q(Z)$ and minimize $KL(Q || P)$. Why do we not minimize $KL(P || Q)$ instead?
0
votes
1answer
14 views

Why do we call the set of latent variables a “family” in variational inference?

we posit a family of approximate densities Q. This is a set of densities over the latent variables. Then, we try to find the member of that family that minimizes the Kullback-Leibler(KL) divergence ...
2
votes
1answer
68 views

Variational Autoencoder, understanding this diagram

I'm not an ML scientist, but I'm trying to understand how variational autoencoder works. I'll take as reference the following diagram, which it couldn't be used for backpropagation as includes a ...
1
vote
0answers
87 views

Can ADVI (Variational Inference) Induce Weak Multi Modality in a system with Uniform Priors, if a Gaussian Variational Family is Used

Question Set Up If I have a weakly multi modal (see below in the edit) target posterior distribution which I am aiming to approximate using ADVI (Automatic Differentiation Variational Inference) with ...
5
votes
2answers
49 views

What is the relation between “conjugate priors” and the approximate inference?

I know that "conjugate prior" is to help us calculate the the denominator of the Bayes formula(to make the calculations easier). And I just learnt to approximate the inference by mean field ...
1
vote
1answer
302 views

Variance of evidence lower bound(ELBO) loss function

When using Bayesian optimisation in a neural network our loss function is equal to: Here the first term is the KL divergence between the approximate and true posteriors. The second term is the ...
0
votes
0answers
14 views

Understanding the inference procedure used in Latent Dirichlet Allocation

I am trying to get a high-level understanding of the inference and parameter estimation procedure of the original Latent Dirichlet Allocation (LDA) paper. Since I'm not too familiar with Bayesian ...
0
votes
1answer
121 views

Meaning of expectation with respect to a function?

This might be a trivial question, but I've come across a paper where some expectation is said to be taken with respect to some pdf. See example: How am I to interpret this, and is there some ...
2
votes
2answers
58 views

Interpreting mixture of Gaussians (Variational Inference)

I've recently stated reading about mixture models and variational inference in this excellent paper, but I'm having troubles dissecting the models described, and have a couple of questions. Please see ...
0
votes
1answer
47 views

Transformation of Uniform Distribution to Real Number Line in ADVI

In the Automatic Differentiation Variational Inference (ADVI) paper, the authors claim to solve the VI problem in a transformed parameter space, which is over $\mathbb{R}$, in order to simplify the ...
1
vote
1answer
56 views

What similarity do VAE encodings have?

Say I am training a VAE on MNIST digit data and it learns to reconstruct the digit images. What will the low-dimensional encodings 'z' have in common between shared classes? Will the distances between ...
2
votes
1answer
25 views

Why are exogenous variables not used in inference/recognition networks?

I have been working lately a lot on amortized variational inference. That is, doing variational inference using neural networks to approximate a variational distribution (such as in Kingma and Welling ...
1
vote
0answers
36 views

Differentiation of Expectation w.r.t distribution

I was looking into Coordinate Ascent Variational Inference formula and came across a step where we need to find the argmax of q($z_j$) for the equation given below. I am not sure how the ...
1
vote
1answer
29 views

In Variational Auto Encoders (VAEs), why is the variance predicted by the encoder lower for noise images?

I am training a VAE on some images, and I want to have some sort of certainty quantifier. Given an input image, the encoder predicts mean and variance vectors, so naturally I thought that the variance ...
4
votes
1answer
245 views

What is the idea behind Bayes By Backprop?

Having looked through the internet and the paper, I find Bayes by Backprop very unaccesible for my intermediate understanding of variational inference. Most online guides also lack some explaining ...
0
votes
1answer
41 views

How do I model the chaotic behaviour(like the sequence from Lorenz attractor) in a stochastic sense?

Recently, I encountered a difficulty of prediction Lorenz attractor by using a GRU. (See the code from here.) I think that it's inevitable since the original system, i.e. Lorenz equation, is too ...
0
votes
0answers
34 views

Lognormal VAE Formulation

I'm looking at the following implementation of a VAE: https://github.com/jmtomczak/vae_vpflows/blob/master/models/VAE.py KL divergence is implemented as: ...
1
vote
1answer
157 views

Variational Autoencoder - How many Normal Distributions for Posterior

I am currently reading about variational autoencoders. Some of the papers I've read are: Tutorial on Variational Autoencoders by Doersch: https://arxiv.org/abs/1606.05908 Auto-Encoding Variational ...
3
votes
1answer
48 views

How does variational inference fit in the big picture of inference?

Apologise for the clickbaity title, but it is difficult to frame this question in a single sentence. Also, the practicality of variational inference is very clear: intractable posteriors; intractable ...
0
votes
1answer
23 views

Notation for conditional within a probability distribution

I'm reading an excellent tutorial on variational autoencoders by Carl Doersch. However, he uses the following notation to define the generative distribution: $$ P(X|z;\theta) = N(X|f(z;\theta), \...
1
vote
1answer
41 views

A technical question about the reparametrisation trick

I was reading this post which enlightened me about the technicalities of the reparametrisation trick, but I only get the intuition of this equivalent transform and I'm not sure why it is true: $$𝐸_𝑞[...
1
vote
0answers
45 views

Is Variational Bayes (VB) and Mean-Field Approximation Useful In practice

I have just had a course in Bayesian Inference, and I am left puzzled about what method should I actually use in practice. Assume I have a multivariate model with multiple parameters $\theta$, where ...
5
votes
1answer
85 views

What is meant by 'Black box variational inference'?

I'm aware of the topic of variational inference (VI) however I'm not really sure what Black box VI is? In particular I am watching a video by David Blei titled Black box variational inference and on ...
2
votes
0answers
38 views

How to characterize the effect of $(\textrm{Diag}(\Sigma^{-1}))^{-1}$ badly approximating $\textrm{Diag}(\Sigma)$

I have an almost singular covariance matrix $\Sigma\in\mathbb{R}^{n\times n}$ that has a few large eigenvalues, followed by many many comparatively very small ev's. If I were to try to approximate ...
0
votes
0answers
27 views

Can this be simplified $\mathbb{E}_{q(\vec{z} \mid \vec{x})}\left[ \log {p(\vec{x} \mid \vec{z})}\right]$?

Assume that $p$ and $q$ are two distributions and $x$ and $z$ are two random variables. Can the following term (which appears in the paper Auto-Encoding Variational Bayes) be further simplified? $$\...
4
votes
1answer
224 views

Why is random sampling a non-differentiable operation?

This answer states that we cannot back-propagate through a random node. So, in the case of VAEs, you have the reparametrisation trick, which shifts the source of randomness to another variable ...
0
votes
1answer
227 views

why is VAE reconstruction loss equal to MSE loss

At which situations does reconstruction loss of VAE equals MSE loss between input and reconstructed output? Other answers where not complete!
3
votes
0answers
57 views

Introduction to Variational Bayesian methods?

I am interested in learning about Variational Bayesian methods. I understand the general idea, explained in Wiki, where the aim is to approximate a posterior using a more tractable distribution, in ...
0
votes
0answers
35 views

What is a sparse Gaussian process?

In the paper Junction Tree Variational Autoencoder for Molecular Graph Generation, section 3.2, the authors state that they train a sparse Gaussian process to predict a chemical property, $y(m)$, of a ...
2
votes
1answer
216 views

VAE: why we do not sample again after decoding and before reconstruction loss?

In many of the VAE schematics and in the original paper, a sampling step is present after decoding and before the reconstruction loss as shown in the image below. The image comes from Stanford CS321n. ...
2
votes
1answer
94 views

Why aren't auto-encoders also considered generative models?

Auto-encoders (AEs) are composed of an encoder and a decoder (often represented by a neural network). The encoder produces a vector representation $z$ of its input $x$ (e.g. an image). The decoder ...
0
votes
0answers
28 views

How can we cast an optimisation problem as an inference problem?

The main idea of variational methods is to cast inference as an optimisation problem. In the paper Junction Tree Variational Autoencoder for Molecular Graph Generation, the authors state that the ...
2
votes
1answer
254 views

What is the relation between ELBO and SGVB?

Evidence lower bound (ELBO) can be minimised, so that to find the most appropriate approximative distribution of the target distribution, which is equivalent to the maximisation of the corresponding ...
0
votes
0answers
63 views

log uniform prior vs gaussian prior to compute KL-divergence in Variational Inference

The optimal value of variational parameters can be found by maximization of the variational lower bound: In some papers, we see that they have proposed log uniform prior and tried to approximate the ...