Questions tagged [variational-bayes]

Variational Bayesian methods approximate intractable integrals found in Bayesian inference and machine learning. Primarily, these methods serve one of two purposes: Approximating the posterior distribution, or bounding the marginal likelihood of observed data.

Filter by
Sorted by
Tagged with
4
votes
1answer
131 views

ELBO maximization with SGD

In cases such as Gaussian mixture models, there's is no closed-term solution for the original likelihood maximization. Maximizing the ELBO, however, does have analytical update formulas (i.e. formulas ...
0
votes
0answers
16 views

Mean-field approximation of a bivariate Gaussian

In their overview paper on variational inference (https://arxiv.org/pdf/1601.00670.pdf), Blei et al. show a contour of a two-dimensional Gaussian (Fig. 1, see below) and note that ‘the optimal mean-...
0
votes
1answer
29 views

Reparameterization trick for exponential distribution

Is there way to generate Exponential(lambda) distributed samples via a reparameterization trick? As in: Reparameterization trick for gamma distribution And also: How does the reparameterization ...
1
vote
1answer
33 views

Why do variational autoencoders not find the “best” latent variables?

From my understanding: Variational autoencoders sample the latent variables $y$ using a proposal distribution $q$ of the observed variables $x$. The objective is that the decoder $p$ applied to $y$ ...
2
votes
2answers
40 views

Bishop derivation completing the square in variational inference

I don't understand the derivation on page 467. Bishop says: Given the optimal factor $q_1^*(z_1)$ \begin{equation} ln~q_1(z_1) = -\frac{1}{2} z_1^2 \Lambda_{11} + z_1 \mu_1 \Lambda_{11} - z_1 \...
0
votes
1answer
17 views

Derivation of the Objective Function for Expectation Propagation

I was reading Expectation Propagation As A Way Of Life and the original paper by Minka Expectation Propagation for Approximate Bayesian Inference and they both say that a fixed point of the EP ...
1
vote
1answer
26 views

How do Variational Autoencoders use Negative Log Likelihood/Cross entropy on real valued outputs?

When training a Variational Autoencoder, the function being maximised is the expected lower bound: $$ \mathscr{L}(\boldsymbol{\theta}, \phi; \mathbf{x}^{(i)}) = -D_{KL}\left(q_{\phi}(\mathbf{z}|\...
0
votes
0answers
18 views

Artificially increase sample size with VAE samples

I want to apply a statistic (KL) on (moderate) high dimensional data and therefore need a lot of data. As its not given, I want to create additional samples that are alike the original data but not ...
1
vote
1answer
31 views

Outlier Detection Using VAE Latent Space

I am trying to perform outlier detection using VAE. Before I was performing the same task using normal autoencoder and I used reconstruction error. I trained the network, then I passed new samples as ...
1
vote
1answer
58 views

Variational autoencoder won't work for toy dataset (mixture of Gaussians)

I wish to use a Variational autoencoder (VAE) as a generator for a multivariate distribution which originates from a graphical model - e.g. samples from a Bayesian Network (I have my reasons...). I ...
1
vote
0answers
24 views

a question about the ELBO surgery in VAE

In the paper "ELBO surgery: yet another way to carve up the variational evidence lower bound", Hoffman et al, propose to rewrite the ELBO, say $KL(q(z|x)||p(z))=KL(q(z)||p(z))+ I (x;z)$. In the ...
0
votes
1answer
26 views

REINFORCE for VAE Implementation Question

I want to compute the VAE loss through REINFORCE since my model's decoder is a deterministic program and is non-differentiable. The only REINFORCE implementation for VAE I was able to find used the ...
0
votes
1answer
17 views

Keras--variational auto-encoder in R studio, which part is defined as Encoder?

This is the example given on VAE, the circle part is something I do not understand. It defined the encoder part as from (X to Z_mean), but my understanding is from(x to Z). Or it just simply does not ...
0
votes
1answer
25 views

Reparametrization trick in VAE at Evaluation time

So I've been trying to implement the Variational Auto-Encoder model of Kigma et.al, but something has been bugging me. While I understand the need for reparametrization trick at training time, the ...
0
votes
1answer
30 views

Why do we penalize individual example divergence in variational autoencoder?

In variational autoencoder, we want to learn a mapping between input space $X$ and latent space $Z$, and $z\in Z$ is related to $x\in X$ with $z\sim MVN(\mu(x), \Sigma(x))$. In addition, we desire ...
1
vote
0answers
22 views

Need some help understanding the factorised posterior in semi-supervised generative modelling

I am having a bit of trouble with the derivation in Kingma's semi-supervised generative modelling paper for the M2-model. The M2 model assumes a probabilistic model where the data $x$ is generated by ...
0
votes
0answers
107 views

VRNN (Variational Recurrent Neural Network) code with Variable Input Lengths on Tensorflow

I have been trying to write VRNN (Variational Recurrent Neural Network: A recurrent latent variable model for sequential data (NIPS 2015)) with variable input data length. My problem is that it is ...
0
votes
1answer
72 views

What is the difference between approximate bayesian computation vs approximate bayesian inference?

What are the main differences between approximate bayesian computation vs approximate bayesian inference? Are they essentially the same? Do they refer to the same of different family of models? My ...
0
votes
0answers
23 views

Evaluating General Variational Inference Local and Global Updates

For coordinate ascent variational inference (CAVI) we can iterate between the "local" parameter updates and the "global" parameter updates. These can be expressed (following [A]) for exponential ...
1
vote
1answer
354 views

Gradients of KL divergence and ELBO for variational inference

When doing variational inference, due to intractability we typically maximize the evidence lower bound (ELBO) instead of minimizing Kullback-Leibler divergence (KLD) between our approximate and exact ...
1
vote
2answers
62 views

Semi-supervised objective function VAE

In Kingma's paper on Semi-supervised learning https://arxiv.org/pdf/1406.5298.pdf, we are shown equations for the ELBO for the semisupervised case, however I am having a hard trying to derive the math ...
0
votes
0answers
41 views

Is “variational approximation” synonymous with “variational inference”?

The title says it all. I am currently reading up on deep generative models, and frequently encounter the term "variational inference" as well as the term "variational approximation" to refer to what ...
0
votes
0answers
25 views

Shouldn't the distribution of alpha be included in the predictive distribution of variational linear regression?

I was reading "pattern recognition and machine learning" book and I see that in the predictive distribution of the bayesian linear regression with variatioanl inference, it uses this equation $p(t|x, ...
1
vote
3answers
431 views

Obtaining VAE reconstruction probability

How does one calculate the reconstruction probability? Let's look at the keras example code from here. Is the reconstruction probability the output of a specific layer, or is it to be calculated ...
1
vote
1answer
37 views

Shouldn't we sample from the output of variational auto-encoder?

I know that the output of the VAE is the parameters of the data. For example: If the data follows normal distribution $X \sim \mathcal{N}(\mu,\sigma)$, the generative network should output $\mu$ and $\...
0
votes
1answer
49 views

Does variational auto-encoder output the variational distribution of the latent variable or the distribution of the input x?

In the simple case of mixture of gaussians(with known variance), we have 2 latent variables $\mu$ and $z$. In the vaiational auto-encoder, we assume that the model is infinite mixture of gaussians. If ...
1
vote
1answer
31 views

Why does variational bayes use $KL(Q || P)$ and not $KL(P || Q)$

In variational Bayes, we approximate the intractable posterior $P(Z | X)$ with a tractable $Q(Z)$ and minimize $KL(Q || P)$. Why do we not minimize $KL(P || Q)$ instead?
0
votes
1answer
18 views

Why do we call the set of latent variables a “family” in variational inference?

we posit a family of approximate densities Q. This is a set of densities over the latent variables. Then, we try to find the member of that family that minimizes the Kullback-Leibler(KL) divergence ...
2
votes
1answer
97 views

Variational Autoencoder, understanding this diagram

I'm not an ML scientist, but I'm trying to understand how variational autoencoder works. I'll take as reference the following diagram, which it couldn't be used for backpropagation as includes a ...
1
vote
0answers
100 views

Can ADVI (Variational Inference) Induce Weak Multi Modality in a system with Uniform Priors, if a Gaussian Variational Family is Used

Question Set Up If I have a weakly multi modal (see below in the edit) target posterior distribution which I am aiming to approximate using ADVI (Automatic Differentiation Variational Inference) with ...
5
votes
2answers
51 views

What is the relation between “conjugate priors” and the approximate inference?

I know that "conjugate prior" is to help us calculate the the denominator of the Bayes formula(to make the calculations easier). And I just learnt to approximate the inference by mean field ...
1
vote
1answer
645 views

Variance of evidence lower bound(ELBO) loss function

When using Bayesian optimisation in a neural network our loss function is equal to: Here the first term is the KL divergence between the approximate and true posteriors. The second term is the ...
0
votes
0answers
17 views

Understanding the inference procedure used in Latent Dirichlet Allocation

I am trying to get a high-level understanding of the inference and parameter estimation procedure of the original Latent Dirichlet Allocation (LDA) paper. Since I'm not too familiar with Bayesian ...
0
votes
1answer
211 views

Meaning of expectation with respect to a function?

This might be a trivial question, but I've come across a paper where some expectation is said to be taken with respect to some pdf. See example: How am I to interpret this, and is there some ...
2
votes
2answers
65 views

Interpreting mixture of Gaussians (Variational Inference)

I've recently stated reading about mixture models and variational inference in this excellent paper, but I'm having troubles dissecting the models described, and have a couple of questions. Please see ...
0
votes
1answer
55 views

Transformation of Uniform Distribution to Real Number Line in ADVI

In the Automatic Differentiation Variational Inference (ADVI) paper, the authors claim to solve the VI problem in a transformed parameter space, which is over $\mathbb{R}$, in order to simplify the ...
1
vote
1answer
71 views

What similarity do VAE encodings have?

Say I am training a VAE on MNIST digit data and it learns to reconstruct the digit images. What will the low-dimensional encodings 'z' have in common between shared classes? Will the distances between ...
2
votes
1answer
26 views

Why are exogenous variables not used in inference/recognition networks?

I have been working lately a lot on amortized variational inference. That is, doing variational inference using neural networks to approximate a variational distribution (such as in Kingma and Welling ...
1
vote
1answer
61 views

Differentiation of Expectation w.r.t distribution

I was looking into Coordinate Ascent Variational Inference formula and came across a step where we need to find the argmax of q($z_j$) for the equation given below. I am not sure how the ...
1
vote
1answer
44 views

In Variational Auto Encoders (VAEs), why is the variance predicted by the encoder lower for noise images?

I am training a VAE on some images, and I want to have some sort of certainty quantifier. Given an input image, the encoder predicts mean and variance vectors, so naturally I thought that the variance ...
4
votes
1answer
377 views

What is the idea behind Bayes By Backprop?

Having looked through the internet and the paper, I find Bayes by Backprop very unaccesible for my intermediate understanding of variational inference. Most online guides also lack some explaining ...
0
votes
1answer
44 views

How do I model the chaotic behaviour(like the sequence from Lorenz attractor) in a stochastic sense?

Recently, I encountered a difficulty of prediction Lorenz attractor by using a GRU. (See the code from here.) I think that it's inevitable since the original system, i.e. Lorenz equation, is too ...
0
votes
0answers
47 views

Lognormal VAE Formulation

I'm looking at the following implementation of a VAE: https://github.com/jmtomczak/vae_vpflows/blob/master/models/VAE.py KL divergence is implemented as: ...
1
vote
1answer
205 views

Variational Autoencoder - How many Normal Distributions for Posterior

I am currently reading about variational autoencoders. Some of the papers I've read are: Tutorial on Variational Autoencoders by Doersch: https://arxiv.org/abs/1606.05908 Auto-Encoding Variational ...
3
votes
1answer
58 views

How does variational inference fit in the big picture of inference?

Apologise for the clickbaity title, but it is difficult to frame this question in a single sentence. Also, the practicality of variational inference is very clear: intractable posteriors; intractable ...
0
votes
1answer
23 views

Notation for conditional within a probability distribution

I'm reading an excellent tutorial on variational autoencoders by Carl Doersch. However, he uses the following notation to define the generative distribution: $$ P(X|z;\theta) = N(X|f(z;\theta), \...
1
vote
1answer
43 views

A technical question about the reparametrisation trick

I was reading this post which enlightened me about the technicalities of the reparametrisation trick, but I only get the intuition of this equivalent transform and I'm not sure why it is true: $$𝐸_𝑞[...
1
vote
0answers
54 views

Is Variational Bayes (VB) and Mean-Field Approximation Useful In practice

I have just had a course in Bayesian Inference, and I am left puzzled about what method should I actually use in practice. Assume I have a multivariate model with multiple parameters $\theta$, where ...
5
votes
1answer
141 views

What is meant by 'Black box variational inference'?

I'm aware of the topic of variational inference (VI) however I'm not really sure what Black box VI is? In particular I am watching a video by David Blei titled Black box variational inference and on ...
2
votes
0answers
38 views

How to characterize the effect of $(\textrm{Diag}(\Sigma^{-1}))^{-1}$ badly approximating $\textrm{Diag}(\Sigma)$

I have an almost singular covariance matrix $\Sigma\in\mathbb{R}^{n\times n}$ that has a few large eigenvalues, followed by many many comparatively very small ev's. If I were to try to approximate ...

1 2 3 4 5