# Questions tagged [variational-bayes]

Variational Bayesian methods approximate intractable integrals found in Bayesian inference and machine learning. Primarily, these methods serve one of two purposes: Approximating the posterior distribution, or bounding the marginal likelihood of observed data.

223 questions
Filter by
Sorted by
Tagged with
0answers
4 views

### Approximate Inference for A Hybrid Dynamic Bayesian Network

In a dynamic bayesian network, if a discrete child has both discrete and continuous parents, how could one do the inference specially the variational inference? For instance, in the below graphical ...
2answers
40 views

### What is the relation between “conjugate priors” and the approximate inference?

I know that "conjugate prior" is to help us calculate the the denominator of the Bayes formula(to make the calculations easier). And I just learnt to approximate the inference by mean field ...
1answer
69 views

### Variance of evidence lower bound(ELBO) loss function

When using Bayesian optimisation in a neural network our loss function is equal to: Here the first term is the KL divergence between the approximate and true posteriors. The second term is the ...
0answers
10 views

### Understanding the inference procedure used in Latent Dirichlet Allocation

I am trying to get a high-level understanding of the inference and parameter estimation procedure of the original Latent Dirichlet Allocation (LDA) paper. Since I'm not too familiar with Bayesian ...
1answer
41 views

### Meaning of expectation with respect to a function?

This might be a trivial question, but I've come across a paper where some expectation is said to be taken with respect to some pdf. See example: How am I to interpret this, and is there some ...
2answers
33 views

### Interpreting mixture of Gaussians (Variational Inference)

I've recently stated reading about mixture models and variational inference in this excellent paper, but I'm having troubles dissecting the models described, and have a couple of questions. Please see ...
1answer
16 views

### Transformation of Uniform Distribution to Real Number Line in ADVI

In the Automatic Differentiation Variational Inference (ADVI) paper, the authors claim to solve the VI problem in a transformed parameter space, which is over $\mathbb{R}$, in order to simplify the ...
1answer
32 views

### What similarity do VAE encodings have?

Say I am training a VAE on MNIST digit data and it learns to reconstruct the digit images. What will the low-dimensional encodings 'z' have in common between shared classes? Will the distances between ...
1answer
25 views

### Why are exogenous variables not used in inference/recognition networks?

I have been working lately a lot on amortized variational inference. That is, doing variational inference using neural networks to approximate a variational distribution (such as in Kingma and Welling ...
0answers
21 views

### Differentiation of Expectation w.r.t distribution

I was looking into Coordinate Ascent Variational Inference formula and came across a step where we need to find the argmax of q($z_j$) for the equation given below. I am not sure how the ...
1answer
22 views

### In Variational Auto Encoders (VAEs), why is the variance predicted by the encoder lower for noise images?

I am training a VAE on some images, and I want to have some sort of certainty quantifier. Given an input image, the encoder predicts mean and variance vectors, so naturally I thought that the variance ...
1answer
63 views

### What is the idea behind Bayes By Backprop?

Having looked through the internet and the paper, I find Bayes by Backprop very unaccesible for my intermediate understanding of variational inference. Most online guides also lack some explaining ...
1answer
38 views

### How do I model the chaotic behaviour(like the sequence from Lorenz attractor) in a stochastic sense?

Recently, I encountered a difficulty of prediction Lorenz attractor by using a GRU. (See the code from here.) I think that it's inevitable since the original system, i.e. Lorenz equation, is too ...
0answers
23 views

### Lognormal VAE Formulation

I'm looking at the following implementation of a VAE: https://github.com/jmtomczak/vae_vpflows/blob/master/models/VAE.py KL divergence is implemented as: ...
1answer
70 views

### Variational Autoencoder - How many Normal Distributions for Posterior

I am currently reading about variational autoencoders. Some of the papers I've read are: Tutorial on Variational Autoencoders by Doersch: https://arxiv.org/abs/1606.05908 Auto-Encoding Variational ...
1answer
39 views

### How does variational inference fit in the big picture of inference?

Apologise for the clickbaity title, but it is difficult to frame this question in a single sentence. Also, the practicality of variational inference is very clear: intractable posteriors; intractable ...
1answer
22 views

0answers
37 views

### Is Variational Bayes (VB) and Mean-Field Approximation Useful In practice

I have just had a course in Bayesian Inference, and I am left puzzled about what method should I actually use in practice. Assume I have a multivariate model with multiple parameters $\theta$, where ...
1answer
61 views

### What is meant by 'Black box variational inference'?

I'm aware of the topic of variational inference (VI) however I'm not really sure what Black box VI is? In particular I am watching a video by David Blei titled Black box variational inference and on ...
0answers
37 views

### How to characterize the effect of $(\textrm{Diag}(\Sigma^{-1}))^{-1}$ badly approximating $\textrm{Diag}(\Sigma)$

I have an almost singular covariance matrix $\Sigma\in\mathbb{R}^{n\times n}$ that has a few large eigenvalues, followed by many many comparatively very small ev's. If I were to try to approximate ...
0answers
25 views

0answers
81 views

### Variational Bayes Lower Bound derivation In Attias paper

I am reading the paper titled "A Variational Bayesian Framework for Graphical Models" by Hagai Attias ( http://www.gatsby.ucl.ac.uk/publications/papers/03-2000.pdf ). I do not follow how Hagai got ...
1answer
56 views

### On evaluating variational autoencoders with prior likelihood and reconstruction error

A common evaluation metric for variational autoencoders (VAEs) is estimating the marginal likelihood of some held-out data, i.e. $p(x)$. This is difficult and often one can only get a lower bound. It'...
1answer
102 views

### Is KL Divergence loss appropriate for generative model? [closed]

I have coordinates data like: ...
0answers
25 views

### Training Variational Autoencoders in two steps

I started to use Variational Autoencoder in a project, (and I have a hard time determining the weight for the reconstruction loss and KL-loss). I have an idea of training the VAE in two steps.: Train ...
1answer
36 views

### Derivative with Reparameterisation Trick

Below is some steps for differentiating a function wrt a set of parameters $\phi$ using the "reparameterisation trick" (Kingma & Welling 2013). However after applying the derivative as follows I ...
1answer
2k views

### KL Divergence loss in variational autoencoders

I was studying VAEs and came accross the Loss function that consists of KL Divergence. I wanted to intuitively make sense of the KL divergence part of the loss function. It would be great if somebody ...
2answers
283 views

### Uncertainty estimation in high-dimensional inference problems without sampling?

I'm working on a high-dimensional inference problem (around 2000 model parameters) for which we are able to robustly perform MAP estimation by finding the global maximum of the log-posterior using a ...
0answers
86 views

### variational lower bound confusion

In this blog describing variational inference under the section KL divergence and ELBO they mention that in the equation $$p(x) = \frac{w(x)}{Z}$$ we can substitute $w$ and $Z$ with: Z = p(x;\...
0answers
15 views

### Does training a VAE online from a nonstationary distribution affect convergence?

For example, using data being sampled from reinforcement learning as the policy improves. If there is an issue, how would we address the issue?
0answers
175 views

### Rao-Blackwellization in variational inference

The Black box VI paper introduces Rao-Blackwellization as a method to reduce the variance of the gradient estimator using score function, in section 3.1. However I don't quite get the basic idea ...
0answers
113 views

### rewriting ELBO to highlight the role of priors

I am reading this paper which rewrites ELBO. I am stuck in verifying the mathematics used for doing the rewriting. Essentially, the paper writes the KL term involved in ELBO as follows (equations 13 ...
1answer
76 views

### What's a mean field variational family?

I'm working through variational Bayesian methods at the moment, and I think I have a grasp of the bigger picture. Where I sometimes have trouble is with the exact details of how it can be implemented. ...
1answer
69 views

### Variational Inference - deriving coordinate update equations for mixture model

I'm currently going through this paper by Blei et. al. that describes the setup and derivation of the coordinate ascent equations for a Gaussian mixture model with K components. I am having some ...