Skip to main content

Questions tagged [variational-bayes]

Variational Bayesian methods approximate intractable integrals found in Bayesian inference and machine learning. Primarily, these methods serve one of two purposes: Approximating the posterior distribution, or bounding the marginal likelihood of observed data.

Filter by
Sorted by
Tagged with
0 votes
0 answers
19 views

Why do we say that we're "predicting" the mean/noise in diffusion models?

In DDPM, ${\tilde\mu}_t$ is the mean of the conditional distribution $q(x_{t-1}|x_t,x_0)$ while the neural network $\mu_\theta$ is modeling a different conditional distribution $p_\theta(x_{t-1}|x_t)$....
Daniel Mendoza's user avatar
2 votes
1 answer
44 views

Why is the forward process referred to as the "ground truth" in diffusion models?

I've seen in many tutorials on diffusion models refer to the distribution of the latent variables induced by the forward process as "ground truth". I wonder why. What we can actually see is ...
Daniel Mendoza's user avatar
5 votes
2 answers
129 views

Theoretical justification for minimizing $KL(q_\phi|p)$ rather than $KL(p|q_\phi)$?

Suppose we have a true but unknown distribution $p$ over some discrete set (i.e. assume no structure or domain knowledge), and a parameterized family of distributions $q_\phi$. In general it makes ...
user56834's user avatar
  • 2,877
0 votes
0 answers
31 views

Prove that variance of log Exp is greater than variance of Exp log

In the context of variational inference, we move from estimating the gradient of: $\log E_{q \sim Q(z|x)} [ \frac{p(z,x)}{q(z|x)} ]$ to: $E_{q \sim Q(z|x)} [ \log \frac{p(z,x)}{q(z|x)} ]$, which is ...
user3180's user avatar
  • 591
0 votes
1 answer
23 views

How to do prediction (evaluate marginal likelihood) in generative latent variable classifier?

The dataset is $\{\boldsymbol x_t,y_t\}$ for $t=1,\dots,T$, where $y_t \in \{0,1\}$. Define a generative latent variable classifier whose plate diagram is shown above. For each data point, a local ...
Kaiwen's user avatar
  • 81
0 votes
0 answers
95 views
+50

Why don’t diffusion models suffer posterior collapse?

In VAEs, posterior collapse occurs when the approximated posterior $q_\theta(z|x)$ becomes the standard Gaussian prior $p(z)$ after training (Lucas et al. 2019). The forward process of diffusion ...
entropy07's user avatar
1 vote
0 answers
42 views

When is variational inference really powerful

I'm learning VI, and specifically Coordinate Ascent Variational Inference, and I'm wondering when we actually get a strong benefit for this method. In particular, the objective is: $$ \min D_{KL}(q(\...
Alberto's user avatar
  • 1,217
0 votes
0 answers
27 views

Conjugate or VB update for normal likelihood with shifted variance

Consider following model with normal likelihood at time $t$. \begin{align*} y_t | \mu_t, \sigma_t^2 \sim \mathcal{N}(\mu_t, \sigma_t^2 + P_t) \end{align*} we would like to find some priors $\mathcal{F}...
StatsyBanksy's user avatar
2 votes
0 answers
73 views

Do discontinuous functions have subgradients also?

Typically, the subgradient is defined for convex functions. And convex functions are continuous. However, DeepMind's VQ-VAE paper defines a model with a discontinuous vector quantization (VQ) layer, ...
MWB's user avatar
  • 1,337
0 votes
0 answers
18 views

Conditions of applications for coordinate ascent variational inference?

In every reference about coordinate ascent variational inference for the mean field family (Chapter 10 Of the book of C.Bishop Pattern recognition and machine learning, or the review article of Blei ...
Pierre Gloaguen's user avatar
1 vote
0 answers
38 views

Using bootstrap for accurate posterior in Variational Bayes

A common well-known issue in Variational Bayes is the variance underestimation of the posterior. Some methods using "sandwich" variance have already been proposed but provide frequentist ...
Mangnier Loïc's user avatar
1 vote
0 answers
39 views

Understanding Variational inference and EM in relation to each other

I have read several answers like here but, somehow I still have a few doubts. I hope to present my understanding and ask a few questions to clear my doubts EM: A maximization maximization algorithm E-...
figs_and_nuts's user avatar
0 votes
0 answers
16 views

How to express a Laplace Distribution as a mixture of an exponential and Normal Distribution [duplicate]

I have seen in different Bayesian papers that a common trick to rewrite a Laplace distribution is to use the fact that the distribution can be expressed as a mixture of an exponential and a Gaussian ...
Mangnier Loïc's user avatar
3 votes
1 answer
90 views

ELBO - Jensen Inequality

In the ELBO derivation: Why do we need the inequality? Why not use the exact equation of the log of expectation? In this video (https://www.youtube.com/watch?v=iL1c1KmYPM0) at minute 34, she says if ...
Nathan G's user avatar
  • 131
0 votes
0 answers
11 views

Why do we need the variance term in SWAG method?

My question is about the SWA-Gaussian paper. I do not really understand why they need the 1/2 factor for the covariance matrix (as underlined in the picture). I understand that it is needed because ...
Mikhail Petrov's user avatar
0 votes
0 answers
19 views

Why is the KL Divergence term present in the Variational Auto Encoder Loss? [duplicate]

I am trying to understand VAEs. A youtube video and a paper that I read about it defined the loss as roughly: $$L=\sum||x-Dec(Enc(x))||_2^2 + D_{KL}(\mathcal N(\mu, \sigma)|\mathcal N(0, 1))$$ The ...
Wolfuryo's user avatar
  • 133
1 vote
0 answers
35 views

Derivatives of matrices

Consider multiple response data from a matrix normal distribution $Y_{n\times m}\sim\text{Normal}(\mathbf{M},\mathbf{U},\mathbf{V})$, where $\mathbf{U}=\mathbf{I}_n$ is the variance among rows and $\...
Chewysplace's user avatar
1 vote
0 answers
22 views

Using MCMC-derived posterior to design variational approximation function

I am trying to fit a hierarchical model that estimates the covariance of some parameters, using the probabilistic programming language pyro. In simulation experiments, I saw that the MCMC generates ...
David Shor's user avatar
2 votes
0 answers
111 views

Derive ELBO for Mixture of Gaussian

I am working through "Variational Inference: A Review for Statisticians" by Blei et al. (see https://arxiv.org/abs/1601.00670) and they illustrate Variational Inference using a Bayesian ...
PSLS's user avatar
  • 41
0 votes
0 answers
28 views

What is the expectation of P(x|z) under distribution of z parameterized by x?

This question stems from Section 2.1 of this VAE tutorial. The problem stated in the paper is to compute the data likelihood using law of total probability: $$ P(X) = \int P(X,z) \,\mathrm dz = \int P(...
Kaiwen's user avatar
  • 81
0 votes
0 answers
24 views

Combining MCMC with Variatonal Inference

I have a Gibbs sampler that is mixing terribly slowly. I have a hunch that if I sample a parameter pair as a single block, it would improve convergence. I tried HMC within Gibbs, but it's also slow. I ...
J. Zeitouni's user avatar
0 votes
1 answer
52 views

Deriving the results in "The Variational Gaussian Approximation Revisited (Opper and Archambeau, 2009)"

I am trying to derive the results in Opper and Archambeau, 2009. In the paper, they show that the variational free energy is the following (Eq. 3.2) $$ \mathcal{F} = \sum_n \langle -\ln \left[p(y_n|...
ItsKalvik's user avatar
0 votes
0 answers
33 views

Why is evidence lower bound concave in the parameters?

The expectation maximization algorithm maximizes the evidence lower bound given by the following equation: $$ELBO = \Sigma_zQ(z)log\frac{p(x,z;\theta)}{Q(z)}$$ Now, the mechanism of the EM algorithm ...
figs_and_nuts's user avatar
0 votes
0 answers
27 views

On the expressivity of latent variable models

Empirically, we have seen that VAEs can approximate very complex distributions. I am interested in knowing if there are any theoretical results showing how expressive latent variable models can be. ...
Saeed Hedayatian's user avatar
0 votes
0 answers
70 views

KL Divergence in VAE [duplicate]

The basic KL-Divergence between two distributions is as: $KL(N(\mu_1,\sigma_1) || N(\mu_2, \sigma_2)) = \log \frac{\sigma_2}{\sigma_1} + \frac{\sigma_1^2 + (\mu_1 - \mu_2)^2}{2 \sigma_2^2} - \frac{1}{...
Martin Perry's user avatar
0 votes
1 answer
154 views

Why does higher dimensional data has higher likelihood?

I am reading about generative models. I came across an example a few times but I cannot come up with an explanation for it. Imagine data is generated according to $p_\text{data}(x)$. It is often said ...
Alf's user avatar
  • 77
0 votes
0 answers
39 views

Why does the reparameterization trick work when some components are still stochastic? [duplicate]

I am trying to understand the reparameterization trick. I got some intuition while looking at this popular question, but I still feel largely confused. I am putting my understanding and doubts here ...
desert_ranger's user avatar
0 votes
0 answers
40 views

Inference of Beta-Bernoulli Distribution

Assume $x_1, x_2, \cdots, x_n$ follows a $Bern(\pi_0)$, Let $y_{ik}$ follows $Beta(\alpha,\beta)$, $i\in \{1,\cdots, n\}$, and $k\in \{1,\cdots, K\}$. Let $z_k$ follows a Bernoulli Distribution with a ...
LAM_MN's user avatar
  • 1
1 vote
0 answers
56 views

Why can Variational Autoencoders (VAEs) approximate arbitrary distributions?

I am trying to reason to myself why is it that VAEs can approximate arbitrary probability distributions even though $q_{\phi}(z|x)$ and $p_{\theta}(x|z)$ are Gaussian. I understand that the parameters ...
Decaying Tails's user avatar
5 votes
1 answer
87 views

Calculation of an optimal variational distribution for covariance parameters in a Bayesian graphical lasso model

Context: I am considering here a variational Bayesian framework where I need to calculate the optimal variational distribution for some covariance parameters. Formally the model can be expressed as: $$...
Mangnier Loïc's user avatar
2 votes
1 answer
88 views

Which is the best way to implement variational inference?

To implement variational inference in a Bayesian model, one essentially has the choice between different approaches that differ in their degree of automation and flexibility: manually deriving update ...
flcello's user avatar
  • 21
1 vote
0 answers
56 views

Initialization of parameters of Bayesian Neural Network

What is the best practice to initialize parameters of the Bayesian Neural Network? I am stuck in a situation where I don’t initialize the parameters and every time I run the variational inference, I ...
newbie's user avatar
  • 225
2 votes
1 answer
237 views

How can be decoder of VAE represent probability distribution p(x|z) eventhough we directly get image as output

How can be decoder of VAE represent probability distribution p(x|z) eventhough we directly get image as output. Also if my current understanding is right than we get same image for same value of z ...
Hamit Des's user avatar
0 votes
2 answers
963 views

How to Resolve Variational Autoencoder (VAE) Model Collapse in Reconstruction Task Using Sensor Data?

I am currently experiencing a suspected model collapse in a Variational Autoencoder (VAE) model I am working with. Below are details on the project setup and the issue at hand: Project Goal: Exploring ...
Bae Browns's user avatar
1 vote
0 answers
30 views

VAE: Noisy decoder output

I'm trying to implement a simple VAE by following several tutorials like this and this. This is the code that I came up with: ...
JosephM's user avatar
  • 11
1 vote
0 answers
48 views

How does Variational Autoencoder approximate the joint probability distribution?

I know that in Variational Inference the idea is to approximate the posterior P(z|x, y) and I know that Variational AutoEncoders (VAEs) use the idea of variational inference through neural network ...
Amir Jalilifard's user avatar
2 votes
1 answer
269 views

Conceptual questions about the proxy distribution in variational inference

I am trying to implement a variational extension of some kind of Bayesian network estimation method. The main goal is to improve speed, since the current method is pretty slow due to MCMC. My question ...
Mangnier Loïc's user avatar
1 vote
0 answers
50 views

Prior term in SGHMC implementation

I am working with SGHMC (Stochastic Gradient Hamiltonian Monte Carlo) models. I found an implimentation of the algorithm in pytorch here. The part of the code that represents momentum variable update (...
Mikhail Petrov's user avatar
3 votes
1 answer
147 views

What is a Student-t VAE ? and how is it different from Gaussian VAE?

I am currently reading https://www.ijcai.org/proceedings/2018/0374.pdf ,this is a research paper based on Student-t Variational Autoencoder for Robust Density Estimation , In this research paper, they ...
Jarvis's user avatar
  • 31
1 vote
0 answers
40 views

Transforming one distribution into another with different support

Consider the following two random variables: A random vector $S$ with law $h$ and support $\mathcal{S}$ and, a random vector $X$ with law $c$ and support $\mathcal{X}$. Assume $\text{Dim}(\mathcal{...
fool's user avatar
  • 2,480
1 vote
1 answer
121 views

Calculating KL divergence with entropy and cross entropy for VAEs

When looking at implementations of VAE's online, specifically the KL divergence loss, the formula used is: $$ KL\hspace{1mm} Loss = -\frac{1}{2}(1+\log{\sigma^2}-\mu^2-\sigma^2) $$ or some variation ...
pyrrosk's user avatar
  • 33
1 vote
0 answers
97 views

In the β-TCVAE paper, can someone help with the derivation (S3) in Appendix C.1?

Paper: Isolating Sources of Disentanglement in VAEs I follow as far as, $$\mathbb{E}_{q(z)}[log[q(z)] = \mathbb{E}_{q(z, n)}[\ log\ \mathbb{E}_{n'\sim\ p(n)}[q(z|n')]\ ]$$ Subsequently, I don't ...
S R's user avatar
  • 33
6 votes
2 answers
1k views

VAE : How is likelihood $p(x|z)$ defined?

Disclaimer : not a strong background in Bayesian statistics. I gather from questions such as this one and this one that in the context of VAEs, we suppose that we know the (form of the ?) prior $p(z)$ ...
Soltius's user avatar
  • 1,366
3 votes
2 answers
360 views

Variational inference : is evidence constant?

I'm studying variational inference (in the context of VAEs), and I'm watching this video at this time point. At this point in the video, the goal of approximating the intractable posterior $p_{\theta}(...
Soltius's user avatar
  • 1,366
6 votes
2 answers
505 views

Is VAE based on mean-field assumption?

According to the slides (section 6), mean-field variational inference (aka. MFVI) assumes the latent variables ($z=\{z_1,..,z_m\}$ are independent from each other, and on top of this assumption, we ...
avocado's user avatar
  • 3,633
0 votes
0 answers
463 views

What is the closed-form of the KL-Divergence between two relaxed Bernoulli distributions?

I've seen in multiple papers that use a relaxation of the Bernoulli distribution as defined in Maddison et. al (here it is referred to as Binary Concrete) and they say that a closed form solution for ...
dannybrig's user avatar
0 votes
1 answer
52 views

How does the decoder predict directly the image in VAEs?

I am reading the VAEs paper Auto-Encoding Variational Bayes. In their loss function: they define reconstruction loss (second RHS term) as the expected value of the log p(x|z) wrt to the posterior of ...
ElPotac's user avatar
  • 61
0 votes
1 answer
190 views

KL divergence between GPs undefined?

In Theorem 1 in Sun et al, Functional Variational Bayesian Neural Networks, 2019, the authors state the the KL divergence between stochastic processes in the supremum over KL divergence between ...
Athere's user avatar
  • 33
2 votes
1 answer
179 views

Mean field approximation convergence

The last sentence of Christopher M. Bishop, Pattern Recognition and Machine Learning Section 10.1.1 Factorized distributions on p.466, states, referring to Equation $(10.9)$, that "Convergence is ...
Hans's user avatar
  • 1,015
1 vote
2 answers
431 views

Deriving the Reparameterization Trick

I am reading about the reparameterization trick from here. It states $\boldsymbol{\epsilon}\sim p(\boldsymbol{\epsilon})$, $\textbf{z}=g_\theta(\boldsymbol{\epsilon},\textbf{x})$, and $$\mathbb{E}_{p_\...
muser's user avatar
  • 145

1
2 3 4 5
8