Stack Exchange Network

Stack Exchange network consists of 174 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [variational-bayes]

Variational Bayesian methods approximate intractable integrals found in Bayesian inference and machine learning. Primarily, these methods serve one of two purposes: Approximating the posterior distribution, or bounding the marginal likelihood of observed data.

0
votes
0answers
4 views

Does training a VAE online from a nonstationary distribution affect convergence?

For example, using data being sampled from reinforcement learning as the policy improves. If there is an issue, how would we address the issue?
4
votes
0answers
40 views

Rao-Blackwellization in variational inference

The Black box VI paper introduces Rao-Blackwellization as a method to reduce the variance of the gradient estimator using score function, in section 3.1. However I don't quite get the basic idea ...
0
votes
0answers
89 views

rewriting ELBO to highlight the role of priors

I am reading this paper which rewrites ELBO. I am stuck in verifying the mathematics used for doing the rewriting. Essentially, the paper writes the KL term involved in ELBO as follows (equations 13 ...
3
votes
1answer
55 views

What's a mean field variational family?

I'm working through variational Bayesian methods at the moment, and I think I have a grasp of the bigger picture. Where I sometimes have trouble is with the exact details of how it can be implemented. ...
0
votes
0answers
16 views

Variational Inference - deriving coordinate update equations for mixture model

I'm currently going through this paper by Blei et. al. that describes the setup and derivation of the coordinate ascent equations for a Gaussian mixture model with K components. I am having some ...
0
votes
0answers
10 views

advantage of variational autoencoder

I know that VAE is generative model. However it is also used as a dimensionality reduction method. In this case, what are advantages of VAE?? Also I saw that well-applied vae on mnist, and it was ...
0
votes
0answers
27 views

Perplexity calculation in variational neural topic models

I'm looking at this 2016 paper from Miao et al. https://arxiv.org/abs/1511.06038 where they use a variational autoencoder for topic modelling. To evaluate the effectiveness of their model, they use ...
0
votes
1answer
17 views

VAE derivation for Gaussian case

In Appendix B of the VAE paper by Kingma and Welling, they derive the KL divergence for the scenario in which $q(\textbf{z})$ and $p(\textbf{z})$ are both Gaussian. I do not understand this step: $$ \...
1
vote
1answer
57 views

Why is variational Bayesian mixture model an alternative to MCMC? What are the similarities?

Why do people say that a variational Bayesian mixture model could be an alternative to MCMC for clustering? For example see the details here: https://en.wikipedia.org/wiki/Variational_Bayesian_method. ...
0
votes
0answers
24 views

Viterbi Algorithm vs Maximum of Variational Posterior for HMM

I have a HMM with observed values $x$ and latent values $z$, upon which I've performed variation inference to get an approximate posterior distribution $q(z|x)$. If I want to calculate a "most likely ...
0
votes
0answers
23 views

Approximation of the upper bound on the expectation of log sum of exponentials

I am having some trouble replicating the results in Guillaume Bouchard's paper, Efficient Bounds for the Softmax Function and Applications to Approximate Inference in Hybrid Models, where he discusses ...
2
votes
0answers
11 views

Estimated Marginal Likelihood in Variational Autoencoder

In Auto-Encoding Variational Bayes Appendix D, the author proposed an accurate marginal likelihood estimator when the dimensionality of latent space is low (<5). $$p_{\mathbf{\theta}}(\mathbf{x}^{(...
1
vote
1answer
22 views

Prerequisites for Wasserstein GAN/Autoencoder

Can someone who read WGAN/WAE papers and understood Wasserstein part, could you share how you prepared necessary Optimal Transport background? The mentioned papers seem little tough if you don't have ...
0
votes
0answers
15 views

Using LDA on sentences of speeches

I can not find a thread or question on the internet which matches my particular case. I want to know whether my approach is fine. I want to compare the sentiment (tone) of particular topics in ...
0
votes
1answer
47 views

Why does my product between normal and gamma distributions not have the expected shape?

I have implemented variational inference according to the model presented in Bishop's Pattern Recognition and Machine Learning (equations (10.21) - (10.30)). The VI algorithm gives me parameters to ...
2
votes
1answer
128 views

Computing KL divergence in loss function of Bayesian neural networks

Hi I am trying to understand how the loss function for Bayesian Neural Networks (BNN) is computed. In the TensorFlow documentation they illustrate a BNN in practice where they train the network to ...
3
votes
1answer
57 views

variational inference derivation

According to this lecture note, Eq. 25 gives the coordinate ascent update for latent variable $z_k$ as follows $$q^*(z_k)\propto\exp(E_{-k}[\log{p(z_k,Z_{-k},x)}])$$ and I understand the derivation ...
0
votes
1answer
67 views

VAE sampling during test time [duplicate]

On page 11 of this VAE tutorial it is said that new samples of the data distribution X can be found by plugging z ~ N(0, I) into the Decoder P. I don't understand why this is true. During training, ...
0
votes
1answer
78 views

VAE with mixture of gaussian prior

I try to understand this paper where they try to use a mixture of Gaussian as a prior, instead of the standard gaussian. There are several things unclear to me though: They say that they set $\pi_k = ...
1
vote
1answer
51 views

Understand VAE with VamPrior

I am currently reading this paper. The authors propose to use as an prior this expression: $$ p_\lambda(z) = \frac{1}{K} \sum^K_{k=1} q_\phi (z\mid u_k) $$ where $q_\phi$ is the encoder, and $u_k$ is ...
2
votes
1answer
34 views

KL-Diverence of Q(z|X) and P(z) in Variational Autoencoder (VAE)

I aim to understand how $D_{KL}[Q(z | X) || P(z)]$ can be converted to $\frac{1}{2} \sum_{k} (\Sigma(X) + \mu^{2}(X) - 1 - \log \Sigma(X))$, where $k$ is the dimension of the Gaussian distribution. ...
1
vote
1answer
53 views

Using step function as activation function in the final layer

I am using variational autoencoders as machine learning algorithm. My input data are images/matrices that represent user interface layouts or how the HTML page will be divided. I am thinking to ...
0
votes
1answer
65 views

how to calculate loglikelihood for VAE/VQVAE

I asked this question on /r/MLQuestions aswell. Although similar questions have been asked a few times here on reddit and elsewhere, I'm still unclear on how one would calculate the log-likelihood of,...
2
votes
2answers
69 views

Lower-bound on covariance estimated via Laplace approximation?

I think when a posterior is approximated to be multivariate normal as in Laplace approximation, the covariance matrix is taken to be the negative inverse Hessian evaluated at the log-posterior maximum,...
3
votes
1answer
95 views

Is the optimization of the Gaussian VAE well-posed?

In a Variational Autoencoder (VAE), given some data $x$ and latent variables $t$ with prior distribution $p(t) = \mathcal{N}(t \mid 0, I)$, the encoder aims to learn a distribution $q_{\phi}(t)$ that ...
0
votes
1answer
117 views

ELBO interpretation in Variational Autoencoder (VAE) for anomaly detection

How to interpret different ELBO values when checking anomaly detection possibilities of VAE model on different "testing" datasets? The higher the ELBO value of the model when testing it on different ...
0
votes
1answer
43 views

Deterministic Decoder for Variational Bayes Autoencoder or RNN

I was wondering if it is possible to use just deterministic function in the output layer of a variational Bayes autoencoder or RNN? Most of papers that I have read are using Gaussian Mixture Models (...
1
vote
1answer
33 views

Do we consider inputs to a VAE to be samples from p(x) and outputs to be samples from p'(x|z) or are they probabilities of x?

In a Variational Autoencoder we say that the encoder and decoder networks model p(z | x) and p(x | z) respectively as seen in the image below: My question is -- is the output of the decoding layer a ...
0
votes
1answer
78 views

Variational Inference: Ising Model

I am self learning Variational Inference. Currently I am reading the chapter 21 book from Murphy 1 and trying to understand the Ising model (21.3.2). The Ising model here is used as denoising ...
0
votes
1answer
164 views

Discrete Random Variables and Deep Generative Models - Why Gumbel-Softmax is needed?

I am reading this 2014 NIPS paper on deep generative models and their application to latent discrete random variables, and this 2017 ICLR paper on Gumbel-Softmax. I essentially don't understand why we ...
3
votes
1answer
59 views

How to choose an appropriate variational distribution?

I work in deep learning research and I am trying to learn how to use variational inference in order to approximate a posterior over the learned weights. I have looked extensively at Yarin Gal's ...
0
votes
1answer
22 views

Using the Expected value of the log as a score for the anomaly detection instead of just the expected value

While dealing with anomaly detection using a probabilistic model I need to compute the probability of an example coming out of the model I built. More specifically: If $p(X)$ is the model I built and ...
0
votes
1answer
103 views

Defining ELBO in Variational Inference with 3 random variables

I am reading this paper, and having a hard time understanding one of the derivations. It is probably more of a stat question. The context is, having three random variables $x,y,z$, we would want to ...
1
vote
1answer
90 views

If Bayesian approaches are better than frequentist then how can it be as practical?

In a textbook Probability Theory: The Logic of Science written by E. T. Jaynes and others, on page 13 it reads that: For many years, there has been controversy over ‘frequentist’ versus ‘...
0
votes
0answers
25 views

Variational Auto-encoder for supervised learning

It seems that variational auto-encoders (VAE) has become one of the most popular technique for generative modeling. However, is it possible to use variational auto encoders for discriminative ...
1
vote
1answer
128 views

Variational Inference: Computation of ELBO and CAVI algorithm

I am reading/studying this paper 1 and got confused with some expressions. It might be basic for many of you, so my apologizes. In the paper the following prior model is assumed: $\mu_k \sim \...
4
votes
1answer
87 views

When generating samples using variational autoencoder, we decode samples from $N(0,1)$ instead of $\mu + \sigma N(0,1)$

Context: I'm trying to understand the use of variational autoencoders as generators. My understanding: During training, for an input point $x_i$ we want to learn latent $\mu_i$ and $\sigma_i$ and ...
2
votes
2answers
313 views

Reparameterization trick for gamma distribution

I am reading the work of Welling on Vartiational Auto-Encoders (VAE), and wonder if there is any way to generate Gamma distributed samples via a similar reparametrization? The idea of ...
0
votes
1answer
249 views

Is the output of a variational autoencoder meant to be a distribution that can be sampled, or a sample directly?

It is difficult to ask this question succinctly in the title, so let me explain. From all the examples of VAEs I have seen, there seem to be 2 approaches used to implement them. In these, ...
0
votes
0answers
25 views

Derivation of ELBO upon the Existence of Conditional Latent Variable Model

I am reading the recently published paper from DeepMind, "Neural Scene Representation and Rendering" and especially its "Supplementary Materials". Following is the page 1 and it's pretty hard for ...
1
vote
0answers
13 views

How to obtain the functional derivative of variational distribution?

Referring to , I want to know how to derive the parameters for the variational distribution, in the Bayesian inference section of the paper. I know how to derive, but I don't know how to deduct the ...
1
vote
0answers
28 views

VAE reconstruction vs generation

I have been reading theory on VAE and trying to understand the loss function. The loss function is made up of Reconstruction loss and KL divergence term. What happens when you have good generated ...
1
vote
1answer
128 views

Vae sampling from prior vs posterior

I have been reading the original VAE paper,Auto-Encoding Variational Bayes. In VAE, when generating samples, why do we sample from prior instead of the learned variational posterior(Fig 5 in the paper)...
2
votes
0answers
230 views

Vae reconstruction loss

I have implemeted a Variational Autoencoder(VAE) with a prior different from unit gaussian. This gives me extremely sharp reconstructions compared to normal VAE(small values for reconstruction loss). ...
1
vote
0answers
36 views

Efficient approximate marginal inference in variational auto-encoder

In Auto-Encoding Variational Bayes, authors mentioned that they proposed a solution to "Efficient approximate marginal inference of the variable $x$". I read through the paper and appendix, now ...
0
votes
0answers
21 views

Analytical solution of $D_{KL}(q_\phi (z) || p_\theta (z))$

In appendix of "Auto-Encoding Variational Bayes" by Kingma & Welling, they provide a solution of $D_{KL}(q_\phi (z) || p_\theta (z))$ for Gaussian case. I feel confused about the notation for the ...
3
votes
0answers
43 views

Generative Adversarial Network and Variational Autoencoders for Independent Component Analysis?

Background: I'm working on a model for independent component analysis (ICA) that is based on a methodology similar to GANs and VAEs. What I'm having trouble understanding is how the choice of the loss ...
0
votes
0answers
36 views

Implementing VAE with initial parameter values

I have been trying to implement a Variational Autoencoder. For VAE, we have a unit Gaussian prior and we infer the posterior using the encoder which gives us mean and logvar. I was wondering if ...
1
vote
0answers
99 views

How do we train variational autoencoders in practice?

I learnt that the objective function of a VAE is given by the RHS of the equation $$\ln p(x_n)-KL(q(z|x_n)\Vert p(z|x_n)) = \Bbb E_{z \sim q(z|x_n)}(\ln p(x_n|z))-KL(q(z|x_n) \Vert p(z))$$ in which $...
2
votes
1answer
296 views

Sampling z in VAE

How many times do we sample from $Q(z|x)$ in a Variational Autoencoder? Let’s say that the autoencoder input $x$ is a single image 28x28 pixels - and $Z$ is is a one dimensional distribution. Then, ...