Questions tagged [variational-bayes]

Variational Bayesian methods approximate intractable integrals found in Bayesian inference and machine learning. Primarily, these methods serve one of two purposes: Approximating the posterior distribution, or bounding the marginal likelihood of observed data.

Filter by
Sorted by
Tagged with
0 votes
0 answers
11 views

Conditions of applications for coordinate ascent variational inference?

In every reference about coordinate ascent variational inference for the mean field family (Chapter 10 Of the book of C.Bishop Pattern recognition and machine learning, or the review article of Blei ...
Pierre Gloaguen's user avatar
1 vote
0 answers
33 views

Using bootstrap for accurate posterior in Variational Bayes

A common well-known issue in Variational Bayes is the variance underestimation of the posterior. Some methods using "sandwich" variance have already been proposed but provide frequentist ...
Mangnier Loïc's user avatar
1 vote
0 answers
29 views

Understanding Variational inference and EM in relation to each other

I have read several answers like here but, somehow I still have a few doubts. I hope to present my understanding and ask a few questions to clear my doubts EM: A maximization maximization algorithm E-...
figs_and_nuts's user avatar
0 votes
0 answers
14 views

How to express a Laplace Distribution as a mixture of an exponential and Normal Distribution [duplicate]

I have seen in different Bayesian papers that a common trick to rewrite a Laplace distribution is to use the fact that the distribution can be expressed as a mixture of an exponential and a Gaussian ...
Mangnier Loïc's user avatar
3 votes
1 answer
75 views

ELBO - Jensen Inequality

In the ELBO derivation: Why do we need the inequality? Why not use the exact equation of the log of expectation? In this video (https://www.youtube.com/watch?v=iL1c1KmYPM0) at minute 34, she says if ...
Nathan B's user avatar
  • 131
0 votes
0 answers
11 views

Why do we need the variance term in SWAG method?

My question is about the SWA-Gaussian paper. I do not really understand why they need the 1/2 factor for the covariance matrix (as underlined in the picture). I understand that it is needed because ...
Mikhail Petrov's user avatar
0 votes
0 answers
18 views

Why is the KL Divergence term present in the Variational Auto Encoder Loss? [duplicate]

I am trying to understand VAEs. A youtube video and a paper that I read about it defined the loss as roughly: $$L=\sum||x-Dec(Enc(x))||_2^2 + D_{KL}(\mathcal N(\mu, \sigma)|\mathcal N(0, 1))$$ The ...
Wolfuryo's user avatar
  • 133
1 vote
0 answers
33 views

Derivatives of matrices

Consider multiple response data from a matrix normal distribution $Y_{n\times m}\sim\text{Normal}(\mathbf{M},\mathbf{U},\mathbf{V})$, where $\mathbf{U}=\mathbf{I}_n$ is the variance among rows and $\...
Chewysplace's user avatar
1 vote
0 answers
22 views

Using MCMC-derived posterior to design variational approximation function

I am trying to fit a hierarchical model that estimates the covariance of some parameters, using the probabilistic programming language pyro. In simulation experiments, I saw that the MCMC generates ...
David Shor's user avatar
2 votes
0 answers
43 views

Derive ELBO for Mixture of Gaussian

I am working through "Variational Inference: A Review for Statisticians" by Blei et al. (see https://arxiv.org/abs/1601.00670) and they illustrate Variational Inference using a Bayesian ...
PSLS's user avatar
  • 41
0 votes
0 answers
27 views

What is the expectation of P(x|z) under distribution of z parameterized by x?

This question stems from Section 2.1 of this VAE tutorial. The problem stated in the paper is to compute the data likelihood using law of total probability: $$ P(X) = \int P(X,z) \,\mathrm dz = \int P(...
Kevin's user avatar
  • 71
0 votes
0 answers
15 views

Combining MCMC with Variatonal Inference

I have a Gibbs sampler that is mixing terribly slowly. I have a hunch that if I sample a parameter pair as a single block, it would improve convergence. I tried HMC within Gibbs, but it's also slow. I ...
J. Zeitouni's user avatar
0 votes
1 answer
45 views

Deriving the results in "The Variational Gaussian Approximation Revisited (Opper and Archambeau, 2009)"

I am trying to derive the results in Opper and Archambeau, 2009. In the paper, they show that the variational free energy is the following (Eq. 3.2) $$ \mathcal{F} = \sum_n \langle -\ln \left[p(y_n|...
ItsKalvik's user avatar
0 votes
0 answers
23 views

Why is evidence lower bound concave in the parameters?

The expectation maximization algorithm maximizes the evidence lower bound given by the following equation: $$ELBO = \Sigma_zQ(z)log\frac{p(x,z;\theta)}{Q(z)}$$ Now, the mechanism of the EM algorithm ...
figs_and_nuts's user avatar
0 votes
0 answers
23 views

On the expressivity of latent variable models

Empirically, we have seen that VAEs can approximate very complex distributions. I am interested in knowing if there are any theoretical results showing how expressive latent variable models can be. ...
Saeed Hedayatian's user avatar
0 votes
0 answers
53 views

KL Divergence in VAE [duplicate]

The basic KL-Divergence between two distributions is as: $KL(N(\mu_1,\sigma_1) || N(\mu_2, \sigma_2)) = \log \frac{\sigma_2}{\sigma_1} + \frac{\sigma_1^2 + (\mu_1 - \mu_2)^2}{2 \sigma_2^2} - \frac{1}{...
Martin Perry's user avatar
0 votes
1 answer
131 views

Why does higher dimensional data has higher likelihood?

I am reading about generative models. I came across an example a few times but I cannot come up with an explanation for it. Imagine data is generated according to $p_\text{data}(x)$. It is often said ...
Alf's user avatar
  • 77
0 votes
0 answers
39 views

Why does the reparameterization trick work when some components are still stochastic? [duplicate]

I am trying to understand the reparameterization trick. I got some intuition while looking at this popular question, but I still feel largely confused. I am putting my understanding and doubts here ...
desert_ranger's user avatar
0 votes
0 answers
32 views

Inference of Beta-Bernoulli Distribution

Assume $x_1, x_2, \cdots, x_n$ follows a $Bern(\pi_0)$, Let $y_{ik}$ follows $Beta(\alpha,\beta)$, $i\in \{1,\cdots, n\}$, and $k\in \{1,\cdots, K\}$. Let $z_k$ follows a Bernoulli Distribution with a ...
LAM_MN's user avatar
  • 1
1 vote
0 answers
41 views

Why can Variational Autoencoders (VAEs) approximate arbitrary distributions?

I am trying to reason to myself why is it that VAEs can approximate arbitrary probability distributions even though $q_{\phi}(z|x)$ and $p_{\theta}(x|z)$ are Gaussian. I understand that the parameters ...
Joel's user avatar
  • 21
4 votes
1 answer
72 views

Calculation of an optimal variational distribution for covariance parameters in a Bayesian graphical lasso model

Context: I am considering here a variational Bayesian framework where I need to calculate the optimal variational distribution for some covariance parameters. Formally the model can be expressed as: $$...
Mangnier Loïc's user avatar
1 vote
0 answers
39 views

Which is the best way to implement variational inference?

To implement variational inference in a Bayesian model, one essentially has the choice between different approaches that differ in their degree of automation and flexibility: manually deriving update ...
flcello's user avatar
  • 11
1 vote
0 answers
51 views

Initialization of parameters of Bayesian Neural Network

What is the best practice to initialize parameters of the Bayesian Neural Network? I am stuck in a situation where I don’t initialize the parameters and every time I run the variational inference, I ...
newbie's user avatar
  • 225
1 vote
1 answer
109 views

How can be decoder of VAE represent probability distribution p(x|z) eventhough we directly get image as output

How can be decoder of VAE represent probability distribution p(x|z) eventhough we directly get image as output. Also if my current understanding is right than we get same image for same value of z ...
Hamit Des's user avatar
0 votes
2 answers
523 views

How to Resolve Variational Autoencoder (VAE) Model Collapse in Reconstruction Task Using Sensor Data?

I am currently experiencing a suspected model collapse in a Variational Autoencoder (VAE) model I am working with. Below are details on the project setup and the issue at hand: Project Goal: Exploring ...
Bae Browns's user avatar
1 vote
0 answers
23 views

VAE: Noisy decoder output

I'm trying to implement a simple VAE by following several tutorials like this and this. This is the code that I came up with: ...
JosephM's user avatar
  • 11
1 vote
0 answers
33 views

How does Variational Autoencoder approximate the joint probability distribution?

I know that in Variational Inference the idea is to approximate the posterior P(z|x, y) and I know that Variational AutoEncoders (VAEs) use the idea of variational inference through neural network ...
Amir Jalilifard's user avatar
2 votes
1 answer
260 views

Conceptual questions about the proxy distribution in variational inference

I am trying to implement a variational extension of some kind of Bayesian network estimation method. The main goal is to improve speed, since the current method is pretty slow due to MCMC. My question ...
Mangnier Loïc's user avatar
1 vote
0 answers
36 views

Prior term in SGHMC implementation

I am working with SGHMC (Stochastic Gradient Hamiltonian Monte Carlo) models. I found an implimentation of the algorithm in pytorch here. The part of the code that represents momentum variable update (...
Mikhail Petrov's user avatar
2 votes
1 answer
115 views

What is a Student-t VAE ? and how is it different from Gaussian VAE?

I am currently reading https://www.ijcai.org/proceedings/2018/0374.pdf ,this is a research paper based on Student-t Variational Autoencoder for Robust Density Estimation , In this research paper, they ...
Jarvis's user avatar
  • 21
0 votes
0 answers
23 views

Differences between Assumed Density Filtering vs Variational Inference

I am new to the concept of Assumed Density Filtering (ADF) but I think that it is very similar to Variational Inference (VI). In VI, we estimate a intractable posterior distribution (p) by finding a ...
Roopali Singh's user avatar
0 votes
0 answers
19 views

Constant terms in variational inference and lower bounds

I have a model for which the lower bound takes the following form $ \log p(x) \geq \mathbb{E}_{recog}[\log \frac{gen}{recog}] + \mathcal{c} $ where recog and gen denote the factorization in the ...
user1571823's user avatar
1 vote
0 answers
33 views

Transforming one distribution into another with different support

Consider the following two random variables: A random vector $S$ with law $h$ and support $\mathcal{S}$ and, a random vector $X$ with law $c$ and support $\mathcal{X}$. Assume $\text{Dim}(\mathcal{...
fool's user avatar
  • 2,440
1 vote
1 answer
95 views

Calculating KL divergence with entropy and cross entropy for VAEs

When looking at implementations of VAE's online, specifically the KL divergence loss, the formula used is: $$ KL\hspace{1mm} Loss = -\frac{1}{2}(1+\log{\sigma^2}-\mu^2-\sigma^2) $$ or some variation ...
pyrrosk's user avatar
  • 43
0 votes
0 answers
18 views

How to put a prior over model parameters?

I am doing a problem in variational inference. I have some data $y$ and I want to understand which distribution it came from. I have the ELBO defined as - $$\text{ELBO}(\phi, D) = \sum_{n=1}^{N} E_{q(...
thedumbkid's user avatar
1 vote
0 answers
91 views

In the β-TCVAE paper, can someone help with the derivation (S3) in Appendix C.1?

Paper: Isolating Sources of Disentanglement in VAEs I follow as far as, $$\mathbb{E}_{q(z)}[log[q(z)] = \mathbb{E}_{q(z, n)}[\ log\ \mathbb{E}_{n'\sim\ p(n)}[q(z|n')]\ ]$$ Subsequently, I don't ...
S R's user avatar
  • 23
0 votes
0 answers
48 views

Graph based variational Autoencoder with variable latent size

I'm trying to build a graph-based Variational-Autoencoder, which should be able to generate graph structures (adjacency matrices). So far, all the papers and models I've seen use a fixed latent vector ...
user3748950's user avatar
5 votes
2 answers
678 views

VAE : How is likelihood $p(x|z)$ defined?

Disclaimer : not a strong background in Bayesian statistics. I gather from questions such as this one and this one that in the context of VAEs, we suppose that we know the (form of the ?) prior $p(z)$ ...
Soltius's user avatar
  • 1,276
2 votes
2 answers
258 views

Variational inference : is evidence constant?

I'm studying variational inference (in the context of VAEs), and I'm watching this video at this time point. At this point in the video, the goal of approximating the intractable posterior $p_{\theta}(...
Soltius's user avatar
  • 1,276
4 votes
3 answers
340 views

Is VAE based on mean-field assumption?

According to the slides (section 6), mean-field variational inference (aka. MFVI) assumes the latent variables ($z=\{z_1,..,z_m\}$ are independent from each other, and on top of this assumption, we ...
avocado's user avatar
  • 3,551
0 votes
0 answers
387 views

What is the closed-form of the KL-Divergence between two relaxed Bernoulli distributions?

I've seen in multiple papers that use a relaxation of the Bernoulli distribution as defined in Maddison et. al (here it is referred to as Binary Concrete) and they say that a closed form solution for ...
dannybrig's user avatar
0 votes
1 answer
43 views

How does the decoder predict directly the image in VAEs?

I am reading the VAEs paper Auto-Encoding Variational Bayes. In their loss function: they define reconstruction loss (second RHS term) as the expected value of the log p(x|z) wrt to the posterior of ...
ElPotac's user avatar
  • 51
0 votes
1 answer
154 views

KL divergence between GPs undefined?

In Theorem 1 in Sun et al, Functional Variational Bayesian Neural Networks, 2019, the authors state the the KL divergence between stochastic processes in the supremum over KL divergence between ...
Athere's user avatar
  • 33
2 votes
1 answer
169 views

Mean field approximation convergence

The last sentence of Christopher M. Bishop, Pattern Recognition and Machine Learning Section 10.1.1 Factorized distributions on p.466, states, referring to Equation $(10.9)$, that "Convergence is ...
Hans's user avatar
  • 995
0 votes
1 answer
261 views

Deriving the Reparameterization Trick

I am reading about the reparameterization trick from here. It states $\boldsymbol{\epsilon}\sim p(\boldsymbol{\epsilon})$, $\textbf{z}=g_\theta(\boldsymbol{\epsilon},\textbf{x})$, and $$\mathbb{E}_{p_\...
muser's user avatar
  • 135
1 vote
0 answers
142 views

Confused about posterior collapse in variational autoencoders

I've been training a $\beta$-VAE with a 5-dimensional latent space on some physics simulation data with 2000 samples. As I increase $\beta$, I notice that an increasing number of the latent variables ...
havocwrecks's user avatar
1 vote
0 answers
42 views

Genetic Algorithm as engine for Variational Inference?

I'm curious if anyone has used, heard of, or otherwise considered using Genetic Algorithms as an engine for Variational Inference (VI)? My understanding of VI is that it's an optimization algorithm, ...
jbuddy_13's user avatar
  • 2,808
1 vote
1 answer
160 views

VAE mean and Standard Deviations are input dependent?

The original presentation of variational autoencoders, VAE assumes the mean $\mu$ and the sd $\sigma$ are functions of the input variable, say $x$. I am studying "Learning Structured Output ...
Toonia's user avatar
  • 13
2 votes
1 answer
402 views

VQ-VAE uniform prior over the latent variables

VQ-VAE propose to make the posterior categorical distribution $q(z|x)$ one-hot = defined as $1$ for the closest embedding $e$ in the codebook to the output of the encoder $z_e(x)$; define a simple ...
ihadanny's user avatar
  • 3,260
2 votes
1 answer
198 views

How to maximize the ELBO in coordinate ascent variational inference

In the lecture by D.Blei: https://www.cs.princeton.edu/courses/archive/fall11/cos597C/lectures/variational-inference-i.pdf Variational inference is explained and he shows how to derive the optimal ...
sam's user avatar
  • 439

1
2 3 4 5
8