Questions tagged [variational-bayes]

Variational Bayesian methods approximate intractable integrals found in Bayesian inference and machine learning. Primarily, these methods serve one of two purposes: Approximating the posterior distribution, or bounding the marginal likelihood of observed data.

Filter by
Sorted by
Tagged with
0
votes
0answers
4 views

Approximate Inference for A Hybrid Dynamic Bayesian Network

In a dynamic bayesian network, if a discrete child has both discrete and continuous parents, how could one do the inference specially the variational inference? For instance, in the below graphical ...
4
votes
2answers
40 views

What is the relation between “conjugate priors” and the approximate inference?

I know that "conjugate prior" is to help us calculate the the denominator of the Bayes formula(to make the calculations easier). And I just learnt to approximate the inference by mean field ...
1
vote
1answer
69 views

Variance of evidence lower bound(ELBO) loss function

When using Bayesian optimisation in a neural network our loss function is equal to: Here the first term is the KL divergence between the approximate and true posteriors. The second term is the ...
0
votes
0answers
10 views

Understanding the inference procedure used in Latent Dirichlet Allocation

I am trying to get a high-level understanding of the inference and parameter estimation procedure of the original Latent Dirichlet Allocation (LDA) paper. Since I'm not too familiar with Bayesian ...
0
votes
1answer
41 views

Meaning of expectation with respect to a function?

This might be a trivial question, but I've come across a paper where some expectation is said to be taken with respect to some pdf. See example: How am I to interpret this, and is there some ...
2
votes
2answers
33 views

Interpreting mixture of Gaussians (Variational Inference)

I've recently stated reading about mixture models and variational inference in this excellent paper, but I'm having troubles dissecting the models described, and have a couple of questions. Please see ...
0
votes
1answer
16 views

Transformation of Uniform Distribution to Real Number Line in ADVI

In the Automatic Differentiation Variational Inference (ADVI) paper, the authors claim to solve the VI problem in a transformed parameter space, which is over $\mathbb{R}$, in order to simplify the ...
1
vote
1answer
32 views

What similarity do VAE encodings have?

Say I am training a VAE on MNIST digit data and it learns to reconstruct the digit images. What will the low-dimensional encodings 'z' have in common between shared classes? Will the distances between ...
2
votes
1answer
25 views

Why are exogenous variables not used in inference/recognition networks?

I have been working lately a lot on amortized variational inference. That is, doing variational inference using neural networks to approximate a variational distribution (such as in Kingma and Welling ...
1
vote
0answers
21 views

Differentiation of Expectation w.r.t distribution

I was looking into Coordinate Ascent Variational Inference formula and came across a step where we need to find the argmax of q($z_j$) for the equation given below. I am not sure how the ...
1
vote
1answer
22 views

In Variational Auto Encoders (VAEs), why is the variance predicted by the encoder lower for noise images?

I am training a VAE on some images, and I want to have some sort of certainty quantifier. Given an input image, the encoder predicts mean and variance vectors, so naturally I thought that the variance ...
4
votes
1answer
63 views

What is the idea behind Bayes By Backprop?

Having looked through the internet and the paper, I find Bayes by Backprop very unaccesible for my intermediate understanding of variational inference. Most online guides also lack some explaining ...
0
votes
1answer
38 views

How do I model the chaotic behaviour(like the sequence from Lorenz attractor) in a stochastic sense?

Recently, I encountered a difficulty of prediction Lorenz attractor by using a GRU. (See the code from here.) I think that it's inevitable since the original system, i.e. Lorenz equation, is too ...
0
votes
0answers
23 views

Lognormal VAE Formulation

I'm looking at the following implementation of a VAE: https://github.com/jmtomczak/vae_vpflows/blob/master/models/VAE.py KL divergence is implemented as: ...
1
vote
1answer
70 views

Variational Autoencoder - How many Normal Distributions for Posterior

I am currently reading about variational autoencoders. Some of the papers I've read are: Tutorial on Variational Autoencoders by Doersch: https://arxiv.org/abs/1606.05908 Auto-Encoding Variational ...
3
votes
1answer
39 views

How does variational inference fit in the big picture of inference?

Apologise for the clickbaity title, but it is difficult to frame this question in a single sentence. Also, the practicality of variational inference is very clear: intractable posteriors; intractable ...
0
votes
1answer
22 views

Notation for conditional within a probability distribution

I'm reading an excellent tutorial on variational autoencoders by Carl Doersch. However, he uses the following notation to define the generative distribution: $$ P(X|z;\theta) = N(X|f(z;\theta), \...
1
vote
1answer
36 views

A technical question about the reparametrisation trick

I was reading this post which enlightened me about the technicalities of the reparametrisation trick, but I only get the intuition of this equivalent transform and I'm not sure why it is true: $$𝐸_𝑞[...
1
vote
0answers
37 views

Is Variational Bayes (VB) and Mean-Field Approximation Useful In practice

I have just had a course in Bayesian Inference, and I am left puzzled about what method should I actually use in practice. Assume I have a multivariate model with multiple parameters $\theta$, where ...
5
votes
1answer
61 views

What is meant by 'Black box variational inference'?

I'm aware of the topic of variational inference (VI) however I'm not really sure what Black box VI is? In particular I am watching a video by David Blei titled Black box variational inference and on ...
2
votes
0answers
37 views

How to characterize the effect of $(\textrm{Diag}(\Sigma^{-1}))^{-1}$ badly approximating $\textrm{Diag}(\Sigma)$

I have an almost singular covariance matrix $\Sigma\in\mathbb{R}^{n\times n}$ that has a few large eigenvalues, followed by many many comparatively very small ev's. If I were to try to approximate ...
0
votes
0answers
25 views

Can this be simplified $\mathbb{E}_{q(\vec{z} \mid \vec{x})}\left[ \log {p(\vec{x} \mid \vec{z})}\right]$?

Assume that $p$ and $q$ are two distributions and $x$ and $z$ are two random variables. Can the following term (which appears in the paper Auto-Encoding Variational Bayes) be further simplified? $$\...
2
votes
1answer
78 views

Why is random sampling a non-differentiable operation?

This answer states that we cannot back-propagate through a random node. So, in the case of VAEs, you have the reparametrisation trick, which shifts the source of randomness to another variable ...
0
votes
1answer
109 views

why is VAE reconstruction loss equal to MSE loss

At which situations does reconstruction loss of VAE equals MSE loss between input and reconstructed output? Other answers where not complete!
3
votes
0answers
54 views

Introduction to Variational Bayesian methods?

I am interested in learning about Variational Bayesian methods. I understand the general idea, explained in Wiki, where the aim is to approximate a posterior using a more tractable distribution, in ...
0
votes
0answers
24 views

What is a sparse Gaussian process?

In the paper Junction Tree Variational Autoencoder for Molecular Graph Generation, section 3.2, the authors state that they train a sparse Gaussian process to predict a chemical property, $y(m)$, of a ...
1
vote
1answer
97 views

VAE: why we do not sample again after decoding and before reconstruction loss?

In many of the VAE schematics and in the original paper, a sampling step is present after decoding and before the reconstruction loss as shown in the image below. The image comes from Stanford CS321n. ...
2
votes
1answer
74 views

Why aren't auto-encoders also considered generative models?

Auto-encoders (AEs) are composed of an encoder and a decoder (often represented by a neural network). The encoder produces a vector representation $z$ of its input $x$ (e.g. an image). The decoder ...
0
votes
0answers
19 views

How can we cast an optimisation problem as an inference problem?

The main idea of variational methods is to cast inference as an optimisation problem. In the paper Junction Tree Variational Autoencoder for Molecular Graph Generation, the authors state that the ...
2
votes
1answer
82 views

What is the relation between ELBO and SGVB?

Evidence lower bound (ELBO) can be minimised, so that to find the most appropriate approximative distribution of the target distribution, which is equivalent to the maximisation of the corresponding ...
0
votes
0answers
35 views

log uniform prior vs gaussian prior to compute KL-divergence in Variational Inference

The optimal value of variational parameters can be found by maximization of the variational lower bound: In some papers, we see that they have proposed log uniform prior and tried to approximate the ...
0
votes
0answers
99 views

Convolutional Conditional Variational Autoencoder Implementation

This may be a rather trivial question, but I am somewhat confused. I have been able to implement a convolutional variational autoencoder. I have also been able to implement a conditional variational ...
0
votes
0answers
24 views

How to understand bayesian inference in the framework of deeplearning?

It is said that $p \left( \theta | y _ { 1 : N } \right) \propto _ { \theta } p \left( y _ { 1 : N } | \theta \right) p ( \theta )$. And $p \left( \theta | y _ { 1 : N } \right)$ is the posterior, $ ...
0
votes
1answer
81 views

Uber's Pyro less accurate than expected on toy example

Trying to understand the pyro example here: https://pyro.ai/examples/svi_part_i.html which starts with a Beta(10,10) prior, adds 10 Bernoulli likelihood datapoints with a 6,4 split. The analytic ...
5
votes
2answers
482 views

Why do we use Gaussian distributions in Variational Autoencoder?

I still don't understand why we force the distribution of the hidden representation of a Variational Autoencoder (VAE) to follow a multivariate normal distribution. Why this specific distribution and ...
1
vote
0answers
50 views

Multivariate Taylor series for moments of a random variable

In the expectation propagation for the generative aspect model, Minka uses Taylor series for the parameter estimation of the topics $p(w\mid a)$ eq 31. I am a little confused in the last equation. He ...
2
votes
1answer
66 views

Equation 10.6 in Bishop book

This is referring to equation 10.6 in Pattern Recognition and Machine Learning by Bishop: $$ L(q) = \int \prod_{i}q_{i} \left[\ln p(X,Z) - \sum \ln q_{i}\right] dZ $$ $$ =\int q_{j}\left[\int \ln p(X,...
0
votes
0answers
81 views

Variational Bayes Lower Bound derivation In Attias paper

I am reading the paper titled "A Variational Bayesian Framework for Graphical Models" by Hagai Attias ( http://www.gatsby.ucl.ac.uk/publications/papers/03-2000.pdf ). I do not follow how Hagai got ...
0
votes
1answer
56 views

On evaluating variational autoencoders with prior likelihood and reconstruction error

A common evaluation metric for variational autoencoders (VAEs) is estimating the marginal likelihood of some held-out data, i.e. $p(x)$. This is difficult and often one can only get a lower bound. It'...
0
votes
1answer
102 views
1
vote
0answers
25 views

Training Variational Autoencoders in two steps

I started to use Variational Autoencoder in a project, (and I have a hard time determining the weight for the reconstruction loss and KL-loss). I have an idea of training the VAE in two steps.: Train ...
0
votes
1answer
36 views

Derivative with Reparameterisation Trick

Below is some steps for differentiating a function wrt a set of parameters $\phi$ using the "reparameterisation trick" (Kingma & Welling 2013). However after applying the derivative as follows I ...
3
votes
1answer
2k views

KL Divergence loss in variational autoencoders

I was studying VAEs and came accross the Loss function that consists of KL Divergence. I wanted to intuitively make sense of the KL divergence part of the loss function. It would be great if somebody ...
9
votes
2answers
283 views

Uncertainty estimation in high-dimensional inference problems without sampling?

I'm working on a high-dimensional inference problem (around 2000 model parameters) for which we are able to robustly perform MAP estimation by finding the global maximum of the log-posterior using a ...
0
votes
0answers
86 views

variational lower bound confusion

In this blog describing variational inference under the section KL divergence and ELBO they mention that in the equation $$p(x) = \frac{w(x)}{Z}$$ we can substitute $w$ and $Z$ with: $$Z = p(x;\...
0
votes
0answers
15 views

Does training a VAE online from a nonstationary distribution affect convergence?

For example, using data being sampled from reinforcement learning as the policy improves. If there is an issue, how would we address the issue?
7
votes
0answers
175 views

Rao-Blackwellization in variational inference

The Black box VI paper introduces Rao-Blackwellization as a method to reduce the variance of the gradient estimator using score function, in section 3.1. However I don't quite get the basic idea ...
0
votes
0answers
113 views

rewriting ELBO to highlight the role of priors

I am reading this paper which rewrites ELBO. I am stuck in verifying the mathematics used for doing the rewriting. Essentially, the paper writes the KL term involved in ELBO as follows (equations 13 ...
3
votes
1answer
76 views

What's a mean field variational family?

I'm working through variational Bayesian methods at the moment, and I think I have a grasp of the bigger picture. Where I sometimes have trouble is with the exact details of how it can be implemented. ...
0
votes
1answer
69 views

Variational Inference - deriving coordinate update equations for mixture model

I'm currently going through this paper by Blei et. al. that describes the setup and derivation of the coordinate ascent equations for a Gaussian mixture model with K components. I am having some ...