Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [variational-bayes]

Variational Bayesian methods approximate intractable integrals found in Bayesian inference and machine learning. Primarily, these methods serve one of two purposes: Approximating the posterior distribution, or bounding the marginal likelihood of observed data.

2
votes
0answers
17 views

How does variational inference fit in the big picture of inference?

Apologise for the clickbaity title, but it is difficult to frame this question in a single sentence. Also, the practicality of variational inference is very clear: intractable posteriors; intractable ...
0
votes
1answer
21 views

Notation for conditional within a probability distribution

I'm reading an excellent tutorial on variational autoencoders by Carl Doersch. However, he uses the following notation to define the generative distribution: $$ P(X|z;\theta) = N(X|f(z;\theta), \...
1
vote
1answer
34 views

A technical question about the reparametrisation trick

I was reading this post which enlightened me about the technicalities of the reparametrisation trick, but I only get the intuition of this equivalent transform and I'm not sure why it is true: $$𝐸_𝑞[...
1
vote
0answers
28 views

Is Variational Bayes (VB) and Mean-Field Approximation Useful In practice

I have just had a course in Bayesian Inference, and I am left puzzled about what method should I actually use in practice. Assume I have a multivariate model with multiple parameters $\theta$, where ...
5
votes
1answer
44 views

What is meant by 'Black box variational inference'?

I'm aware of the topic of variational inference (VI) however I'm not really sure what Black box VI is? In particular I am watching a video by David Blei titled Black box variational inference and on ...
2
votes
0answers
35 views

How to characterize the effect of $(\textrm{Diag}(\Sigma^{-1}))^{-1}$ badly approximating $\textrm{Diag}(\Sigma)$

I have an almost singular covariance matrix $\Sigma\in\mathbb{R}^{n\times n}$ that has a few large eigenvalues, followed by many many comparatively very small ev's. If I were to try to approximate ...
0
votes
0answers
23 views

Can this be simplified $\mathbb{E}_{q(\vec{z} \mid \vec{x})}\left[ \log {p(\vec{x} \mid \vec{z})}\right]$?

Assume that $p$ and $q$ are two distributions and $x$ and $z$ are two random variables. Can the following term (which appears in the paper Auto-Encoding Variational Bayes) be further simplified? $$\...
2
votes
1answer
42 views

Why is random sampling a non-differentiable operation?

This answer states that we cannot back-propagate through a random node. So, in the case of VAEs, you have the reparametrisation trick, which shifts the source of randomness to another variable ...
0
votes
0answers
48 views

why is VAE reconstruction loss equal to MSE loss

At which situations does reconstruction loss of VAE equals MSE loss between input and reconstructed output? Other answers where not complete!
3
votes
0answers
46 views

Introduction to Variational Bayesian methods?

I am interested in learning about Variational Bayesian methods. I understand the general idea, explained in Wiki, where the aim is to approximate a posterior using a more tractable distribution, in ...
0
votes
0answers
18 views

What is a sparse Gaussian process?

In the paper Junction Tree Variational Autoencoder for Molecular Graph Generation, section 3.2, the authors state that they train a sparse Gaussian process to predict a chemical property, $y(m)$, of a ...
1
vote
1answer
59 views

VAE: why we do not sample again after decoding and before reconstruction loss?

In many of the VAE schematics and in the original paper, a sampling step is present after decoding and before the reconstruction loss as shown in the image below. The image comes from Stanford CS321n. ...
1
vote
0answers
37 views

Why aren't auto-encoders also considered generative models?

Auto-encoders (AEs) are composed of an encoder and a decoder (often represented by a neural network). The encoder produces a vector representation $z$ of its input $x$ (e.g. an image). The decoder ...
0
votes
0answers
12 views

How can we cast an optimisation problem as an inference problem?

The main idea of variational methods is to cast inference as an optimisation problem. In the paper Junction Tree Variational Autoencoder for Molecular Graph Generation, the authors state that the ...
2
votes
1answer
40 views

What is the relation between ELBO and SGVB?

Evidence lower bound (ELBO) can be minimised, so that to find the most appropriate approximative distribution of the target distribution, which is equivalent to the maximisation of the corresponding ...
0
votes
0answers
18 views

log uniform prior vs gaussian prior to compute KL-divergence in Variational Inference

The optimal value of variational parameters can be found by maximization of the variational lower bound: In some papers, we see that they have proposed log uniform prior and tried to approximate the ...
0
votes
0answers
54 views

Convolutional Conditional Variational Autoencoder Implementation

This may be a rather trivial question, but I am somewhat confused. I have been able to implement a convolutional variational autoencoder. I have also been able to implement a conditional variational ...
0
votes
0answers
21 views

How to understand bayesian inference in the framework of deeplearning?

It is said that $p \left( \theta | y _ { 1 : N } \right) \propto _ { \theta } p \left( y _ { 1 : N } | \theta \right) p ( \theta )$. And $p \left( \theta | y _ { 1 : N } \right)$ is the posterior, $ ...
0
votes
1answer
49 views

Uber's Pyro less accurate than expected on toy example

Trying to understand the pyro example here: https://pyro.ai/examples/svi_part_i.html which starts with a Beta(10,10) prior, adds 10 Bernoulli likelihood datapoints with a 6,4 split. The analytic ...
4
votes
2answers
187 views

Why do we use Gaussian distributions in Variational Autoencoder?

I still don't understand why we force the distribution of the hidden representation of a Variational Autoencoder (VAE) to follow a multivariate normal distribution. Why this specific distribution and ...
1
vote
0answers
31 views

Multivariate Taylor series for moments of a random variable

In the expectation propagation for the generative aspect model, Minka uses Taylor series for the parameter estimation of the topics $p(w\mid a)$ eq 31. I am a little confused in the last equation. He ...
2
votes
1answer
57 views

Equation 10.6 in Bishop book

This is referring to equation 10.6 in Pattern Recognition and Machine Learning by Bishop: $$ L(q) = \int \prod_{i}q_{i} \left[\ln p(X,Z) - \sum \ln q_{i}\right] dZ $$ $$ =\int q_{j}\left[\int \ln p(X,...
0
votes
0answers
74 views

Variational Bayes Lower Bound derivation In Attias paper

I am reading the paper titled "A Variational Bayesian Framework for Graphical Models" by Hagai Attias ( http://www.gatsby.ucl.ac.uk/publications/papers/03-2000.pdf ). I do not follow how Hagai got ...
0
votes
1answer
38 views

On evaluating variational autoencoders with prior likelihood and reconstruction error

A common evaluation metric for variational autoencoders (VAEs) is estimating the marginal likelihood of some held-out data, i.e. $p(x)$. This is difficult and often one can only get a lower bound. It'...
0
votes
1answer
53 views
1
vote
0answers
23 views

Training Variational Autoencoders in two steps

I started to use Variational Autoencoder in a project, (and I have a hard time determining the weight for the reconstruction loss and KL-loss). I have an idea of training the VAE in two steps.: Train ...
0
votes
1answer
36 views

Derivative with Reparameterisation Trick

Below is some steps for differentiating a function wrt a set of parameters $\phi$ using the "reparameterisation trick" (Kingma & Welling 2013). However after applying the derivative as follows I ...
2
votes
1answer
749 views

KL Divergence loss in variational autoencoders

I was studying VAEs and came accross the Loss function that consists of KL Divergence. I wanted to intuitively make sense of the KL divergence part of the loss function. It would be great if somebody ...
8
votes
2answers
274 views

Uncertainty estimation in high-dimensional inference problems without sampling?

I'm working on a high-dimensional inference problem (around 2000 model parameters) for which we are able to robustly perform MAP estimation by finding the global maximum of the log-posterior using a ...
0
votes
0answers
43 views

variational lower bound confusion

In this blog describing variational inference under the section KL divergence and ELBO they mention that in the equation $$p(x) = \frac{w(x)}{Z}$$ we can substitute $w$ and $Z$ with: $$Z = p(x;\...
0
votes
0answers
13 views

Does training a VAE online from a nonstationary distribution affect convergence?

For example, using data being sampled from reinforcement learning as the policy improves. If there is an issue, how would we address the issue?
7
votes
0answers
157 views

Rao-Blackwellization in variational inference

The Black box VI paper introduces Rao-Blackwellization as a method to reduce the variance of the gradient estimator using score function, in section 3.1. However I don't quite get the basic idea ...
0
votes
0answers
111 views

rewriting ELBO to highlight the role of priors

I am reading this paper which rewrites ELBO. I am stuck in verifying the mathematics used for doing the rewriting. Essentially, the paper writes the KL term involved in ELBO as follows (equations 13 ...
3
votes
1answer
63 views

What's a mean field variational family?

I'm working through variational Bayesian methods at the moment, and I think I have a grasp of the bigger picture. Where I sometimes have trouble is with the exact details of how it can be implemented. ...
0
votes
1answer
62 views

Variational Inference - deriving coordinate update equations for mixture model

I'm currently going through this paper by Blei et. al. that describes the setup and derivation of the coordinate ascent equations for a Gaussian mixture model with K components. I am having some ...
0
votes
1answer
246 views

advantage of variational autoencoder

I know that VAE is generative model. However it is also used as a dimensionality reduction method. In this case, what are advantages of VAE?? Also I saw that well-applied vae on mnist, and it was ...
1
vote
0answers
76 views

Perplexity calculation in variational neural topic models

I'm looking at this 2016 paper from Miao et al. https://arxiv.org/abs/1511.06038 where they use a variational autoencoder for topic modelling. To evaluate the effectiveness of their model, they use ...
1
vote
1answer
49 views

VAE derivation for Gaussian case

In Appendix B of the VAE paper by Kingma and Welling, they derive the KL divergence for the scenario in which $q(\textbf{z})$ and $p(\textbf{z})$ are both Gaussian. I do not understand this step: $$ \...
1
vote
1answer
101 views

Why is variational Bayesian mixture model an alternative to MCMC? What are the similarities?

Why do people say that a variational Bayesian mixture model could be an alternative to MCMC for clustering? For example see the details here: https://en.wikipedia.org/wiki/Variational_Bayesian_method. ...
0
votes
0answers
47 views

Viterbi Algorithm vs Maximum of Variational Posterior for HMM

I have a HMM with observed values $x$ and latent values $z$, upon which I've performed variation inference to get an approximate posterior distribution $q(z|x)$. If I want to calculate a "most likely ...
0
votes
0answers
30 views

Approximation of the upper bound on the expectation of log sum of exponentials

I am having some trouble replicating the results in Guillaume Bouchard's paper, Efficient Bounds for the Softmax Function and Applications to Approximate Inference in Hybrid Models, where he discusses ...
2
votes
0answers
39 views

Estimated Marginal Likelihood in Variational Autoencoder

In Auto-Encoding Variational Bayes Appendix D, the author proposed an accurate marginal likelihood estimator when the dimensionality of latent space is low (<5). $$p_{\mathbf{\theta}}(\mathbf{x}^{(...
2
votes
1answer
47 views

Prerequisites for Wasserstein GAN/Autoencoder

Can someone who read WGAN/WAE papers and understood Wasserstein part, could you share how you prepared necessary Optimal Transport background? The mentioned papers seem little tough if you don't have ...
0
votes
0answers
25 views

Using LDA on sentences of speeches

I can not find a thread or question on the internet which matches my particular case. I want to know whether my approach is fine. I want to compare the sentiment (tone) of particular topics in ...
0
votes
1answer
64 views

Why does my product between normal and gamma distributions not have the expected shape?

I have implemented variational inference according to the model presented in Bishop's Pattern Recognition and Machine Learning (equations (10.21) - (10.30)). The VI algorithm gives me parameters to ...
2
votes
1answer
571 views

Computing KL divergence in loss function of Bayesian neural networks

Hi I am trying to understand how the loss function for Bayesian Neural Networks (BNN) is computed. In the TensorFlow documentation they illustrate a BNN in practice where they train the network to ...
3
votes
1answer
72 views

variational inference derivation

According to this lecture note, Eq. 25 gives the coordinate ascent update for latent variable $z_k$ as follows $$q^*(z_k)\propto\exp(E_{-k}[\log{p(z_k,Z_{-k},x)}])$$ and I understand the derivation ...
0
votes
1answer
265 views

VAE sampling during test time [duplicate]

On page 11 of this VAE tutorial it is said that new samples of the data distribution X can be found by plugging z ~ N(0, I) into the Decoder P. I don't understand why this is true. During training, ...
0
votes
1answer
209 views

VAE with mixture of gaussian prior

I try to understand this paper where they try to use a mixture of Gaussian as a prior, instead of the standard gaussian. There are several things unclear to me though: They say that they set $\pi_k = ...
1
vote
1answer
85 views

Understand VAE with VamPrior

I am currently reading this paper. The authors propose to use as an prior this expression: $$ p_\lambda(z) = \frac{1}{K} \sum^K_{k=1} q_\phi (z\mid u_k) $$ where $q_\phi$ is the encoder, and $u_k$ is ...