Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [variational-bayes]

Variational Bayesian methods approximate intractable integrals found in Bayesian inference and machine learning. Primarily, these methods serve one of two purposes: Approximating the posterior distribution, or bounding the marginal likelihood of observed data.

0
votes
0answers
11 views

Uber's Pyro less accurate than expected on toy example

Trying to understand the pyro example here: https://pyro.ai/examples/svi_part_i.html which starts with a Beta(10,10) prior, adds 10 Bernoulli likelihood datapoints with a 6,4 split. The analytic ...
3
votes
2answers
59 views

Why do we use Gaussian distributions in Variational Autoencoder?

I still don't understand why we force the distribution of the hidden representation of a Variational Autoencoder (VAE) to follow a multivariate normal distribution. Why this specific distribution and ...
1
vote
0answers
16 views

Multivariate Taylor series for moments of a random variable

In the expectation propagation for the generative aspect model, Minka uses Taylor series for the parameter estimation of the topics $p(w\mid a)$ eq 31. I am a little confused in the last equation. He ...
2
votes
1answer
46 views

Equation 10.6 in Bishop book

This is referring to equation 10.6 in Pattern Recognition and Machine Learning by Bishop: $$ L(q) = \int \prod_{i}q_{i} \left[\ln p(X,Z) - \sum \ln q_{i}\right] dZ $$ $$ =\int q_{j}\left[\int \ln p(X,...
0
votes
0answers
43 views

Variational Bayes Lower Bound derivation In Attias paper

I am reading the paper titled "A Variational Bayesian Framework for Graphical Models" by Hagai Attias ( http://www.gatsby.ucl.ac.uk/publications/papers/03-2000.pdf ). I do not follow how Hagai got ...
0
votes
1answer
17 views

On evaluating variational autoencoders with prior likelihood and reconstruction error

A common evaluation metric for variational autoencoders (VAEs) is estimating the marginal likelihood of some held-out data, i.e. $p(x)$. This is difficult and often one can only get a lower bound. It'...
0
votes
1answer
33 views
1
vote
0answers
16 views

Training Variational Autoencoders in two steps

I started to use Variational Autoencoder in a project, (and I have a hard time determining the weight for the reconstruction loss and KL-loss). I have an idea of training the VAE in two steps.: Train ...
0
votes
1answer
30 views

Derivative with Reparameterisation Trick

Below is some steps for differentiating a function wrt a set of parameters $\phi$ using the "reparameterisation trick" (Kingma & Welling 2013). However after applying the derivative as follows I ...
2
votes
1answer
184 views

KL Divergence loss in variational autoencoders

I was studying VAEs and came accross the Loss function that consists of KL Divergence. I wanted to intuitively make sense of the KL divergence part of the loss function. It would be great if somebody ...
8
votes
2answers
214 views

Uncertainty estimation in high-dimensional inference problems without sampling?

I'm working on a high-dimensional inference problem (around 2000 model parameters) for which we are able to robustly perform MAP estimation by finding the global maximum of the log-posterior using a ...
0
votes
0answers
23 views

variational lower bound confusion

In this blog describing variational inference under the section KL divergence and ELBO they mention that in the equation $$p(x) = \frac{w(x)}{Z}$$ we can substitute $w$ and $Z$ with: $$Z = p(x;\...
0
votes
0answers
12 views

Does training a VAE online from a nonstationary distribution affect convergence?

For example, using data being sampled from reinforcement learning as the policy improves. If there is an issue, how would we address the issue?
7
votes
0answers
137 views

Rao-Blackwellization in variational inference

The Black box VI paper introduces Rao-Blackwellization as a method to reduce the variance of the gradient estimator using score function, in section 3.1. However I don't quite get the basic idea ...
0
votes
0answers
101 views

rewriting ELBO to highlight the role of priors

I am reading this paper which rewrites ELBO. I am stuck in verifying the mathematics used for doing the rewriting. Essentially, the paper writes the KL term involved in ELBO as follows (equations 13 ...
3
votes
1answer
60 views

What's a mean field variational family?

I'm working through variational Bayesian methods at the moment, and I think I have a grasp of the bigger picture. Where I sometimes have trouble is with the exact details of how it can be implemented. ...
0
votes
1answer
40 views

Variational Inference - deriving coordinate update equations for mixture model

I'm currently going through this paper by Blei et. al. that describes the setup and derivation of the coordinate ascent equations for a Gaussian mixture model with K components. I am having some ...
0
votes
1answer
91 views

advantage of variational autoencoder

I know that VAE is generative model. However it is also used as a dimensionality reduction method. In this case, what are advantages of VAE?? Also I saw that well-applied vae on mnist, and it was ...
0
votes
0answers
45 views

Perplexity calculation in variational neural topic models

I'm looking at this 2016 paper from Miao et al. https://arxiv.org/abs/1511.06038 where they use a variational autoencoder for topic modelling. To evaluate the effectiveness of their model, they use ...
0
votes
1answer
27 views

VAE derivation for Gaussian case

In Appendix B of the VAE paper by Kingma and Welling, they derive the KL divergence for the scenario in which $q(\textbf{z})$ and $p(\textbf{z})$ are both Gaussian. I do not understand this step: $$ \...
1
vote
1answer
77 views

Why is variational Bayesian mixture model an alternative to MCMC? What are the similarities?

Why do people say that a variational Bayesian mixture model could be an alternative to MCMC for clustering? For example see the details here: https://en.wikipedia.org/wiki/Variational_Bayesian_method. ...
0
votes
0answers
34 views

Viterbi Algorithm vs Maximum of Variational Posterior for HMM

I have a HMM with observed values $x$ and latent values $z$, upon which I've performed variation inference to get an approximate posterior distribution $q(z|x)$. If I want to calculate a "most likely ...
0
votes
0answers
26 views

Approximation of the upper bound on the expectation of log sum of exponentials

I am having some trouble replicating the results in Guillaume Bouchard's paper, Efficient Bounds for the Softmax Function and Applications to Approximate Inference in Hybrid Models, where he discusses ...
2
votes
0answers
17 views

Estimated Marginal Likelihood in Variational Autoencoder

In Auto-Encoding Variational Bayes Appendix D, the author proposed an accurate marginal likelihood estimator when the dimensionality of latent space is low (<5). $$p_{\mathbf{\theta}}(\mathbf{x}^{(...
2
votes
1answer
35 views

Prerequisites for Wasserstein GAN/Autoencoder

Can someone who read WGAN/WAE papers and understood Wasserstein part, could you share how you prepared necessary Optimal Transport background? The mentioned papers seem little tough if you don't have ...
0
votes
0answers
23 views

Using LDA on sentences of speeches

I can not find a thread or question on the internet which matches my particular case. I want to know whether my approach is fine. I want to compare the sentiment (tone) of particular topics in ...
0
votes
1answer
57 views

Why does my product between normal and gamma distributions not have the expected shape?

I have implemented variational inference according to the model presented in Bishop's Pattern Recognition and Machine Learning (equations (10.21) - (10.30)). The VI algorithm gives me parameters to ...
2
votes
1answer
320 views

Computing KL divergence in loss function of Bayesian neural networks

Hi I am trying to understand how the loss function for Bayesian Neural Networks (BNN) is computed. In the TensorFlow documentation they illustrate a BNN in practice where they train the network to ...
3
votes
1answer
66 views

variational inference derivation

According to this lecture note, Eq. 25 gives the coordinate ascent update for latent variable $z_k$ as follows $$q^*(z_k)\propto\exp(E_{-k}[\log{p(z_k,Z_{-k},x)}])$$ and I understand the derivation ...
0
votes
1answer
163 views

VAE sampling during test time [duplicate]

On page 11 of this VAE tutorial it is said that new samples of the data distribution X can be found by plugging z ~ N(0, I) into the Decoder P. I don't understand why this is true. During training, ...
0
votes
1answer
155 views

VAE with mixture of gaussian prior

I try to understand this paper where they try to use a mixture of Gaussian as a prior, instead of the standard gaussian. There are several things unclear to me though: They say that they set $\pi_k = ...
1
vote
1answer
67 views

Understand VAE with VamPrior

I am currently reading this paper. The authors propose to use as an prior this expression: $$ p_\lambda(z) = \frac{1}{K} \sum^K_{k=1} q_\phi (z\mid u_k) $$ where $q_\phi$ is the encoder, and $u_k$ is ...
2
votes
1answer
43 views

KL-Diverence of Q(z|X) and P(z) in Variational Autoencoder (VAE)

I aim to understand how $D_{KL}[Q(z | X) || P(z)]$ can be converted to $\frac{1}{2} \sum_{k} (\Sigma(X) + \mu^{2}(X) - 1 - \log \Sigma(X))$, where $k$ is the dimension of the Gaussian distribution. ...
1
vote
1answer
61 views

Using step function as activation function in the final layer

I am using variational autoencoders as machine learning algorithm. My input data are images/matrices that represent user interface layouts or how the HTML page will be divided. I am thinking to ...
1
vote
1answer
147 views

how to calculate loglikelihood for VAE/VQVAE

I asked this question on /r/MLQuestions aswell. Although similar questions have been asked a few times here on reddit and elsewhere, I'm still unclear on how one would calculate the log-likelihood of,...
2
votes
2answers
78 views

Lower-bound on covariance estimated via Laplace approximation?

I think when a posterior is approximated to be multivariate normal as in Laplace approximation, the covariance matrix is taken to be the negative inverse Hessian evaluated at the log-posterior maximum,...
3
votes
1answer
124 views

Is the optimization of the Gaussian VAE well-posed?

In a Variational Autoencoder (VAE), given some data $x$ and latent variables $t$ with prior distribution $p(t) = \mathcal{N}(t \mid 0, I)$, the encoder aims to learn a distribution $q_{\phi}(t)$ that ...
0
votes
1answer
194 views

ELBO interpretation in Variational Autoencoder (VAE) for anomaly detection

How to interpret different ELBO values when checking anomaly detection possibilities of VAE model on different "testing" datasets? The higher the ELBO value of the model when testing it on different ...
0
votes
1answer
81 views

Deterministic Decoder for Variational Bayes Autoencoder or RNN

I was wondering if it is possible to use just deterministic function in the output layer of a variational Bayes autoencoder or RNN? Most of papers that I have read are using Gaussian Mixture Models (...
1
vote
1answer
48 views

Do we consider inputs to a VAE to be samples from p(x) and outputs to be samples from p'(x|z) or are they probabilities of x?

In a Variational Autoencoder we say that the encoder and decoder networks model p(z | x) and p(x | z) respectively as seen in the image below: My question is -- is the output of the decoding layer a ...
1
vote
1answer
109 views

Variational Inference: Ising Model

I am self learning Variational Inference. Currently I am reading the chapter 21 book from Murphy 1 and trying to understand the Ising model (21.3.2). The Ising model here is used as denoising ...
0
votes
1answer
187 views

Discrete Random Variables and Deep Generative Models - Why Gumbel-Softmax is needed?

I am reading this 2014 NIPS paper on deep generative models and their application to latent discrete random variables, and this 2017 ICLR paper on Gumbel-Softmax. I essentially don't understand why we ...
3
votes
1answer
77 views

How to choose an appropriate variational distribution?

I work in deep learning research and I am trying to learn how to use variational inference in order to approximate a posterior over the learned weights. I have looked extensively at Yarin Gal's ...
0
votes
1answer
23 views

Using the Expected value of the log as a score for the anomaly detection instead of just the expected value

While dealing with anomaly detection using a probabilistic model I need to compute the probability of an example coming out of the model I built. More specifically: If $p(X)$ is the model I built and ...
0
votes
1answer
117 views

Defining ELBO in Variational Inference with 3 random variables

I am reading this paper, and having a hard time understanding one of the derivations. It is probably more of a stat question. The context is, having three random variables $x,y,z$, we would want to ...
1
vote
1answer
107 views

If Bayesian approaches are better than frequentist then how can it be as practical?

In a textbook Probability Theory: The Logic of Science written by E. T. Jaynes and others, on page 13 it reads that: For many years, there has been controversy over ‘frequentist’ versus ‘...
0
votes
0answers
51 views

Variational Auto-encoder for supervised learning

It seems that variational auto-encoders (VAE) has become one of the most popular technique for generative modeling. However, is it possible to use variational auto encoders for discriminative ...
1
vote
1answer
194 views

Variational Inference: Computation of ELBO and CAVI algorithm

I am reading/studying this paper 1 and got confused with some expressions. It might be basic for many of you, so my apologizes. In the paper the following prior model is assumed: $\mu_k \sim \...
4
votes
1answer
191 views

When generating samples using variational autoencoder, we decode samples from $N(0,1)$ instead of $\mu + \sigma N(0,1)$

Context: I'm trying to understand the use of variational autoencoders as generators. My understanding: During training, for an input point $x_i$ we want to learn latent $\mu_i$ and $\sigma_i$ and ...
2
votes
2answers
482 views

Reparameterization trick for gamma distribution

I am reading the work of Welling on Vartiational Auto-Encoders (VAE), and wonder if there is any way to generate Gamma distributed samples via a similar reparametrization? The idea of ...