Questions tagged [variational-bayes]
Variational Bayesian methods approximate intractable integrals found in Bayesian inference and machine learning. Primarily, these methods serve one of two purposes: Approximating the posterior distribution, or bounding the marginal likelihood of observed data.
379
questions
0
votes
0
answers
15
views
On the expressivity of latent variable models
Empirically, we have seen that VAEs can approximate very complex distributions. I am interested in knowing if there are any theoretical results showing how expressive latent variable models can be. ...
0
votes
0
answers
6
views
Variational Diffusion Models (DDPM/MHVAE) -- samples vs distribution parameters [closed]
Background
I've been learning about diffusion models, specifically reading the original de-noising diffusion probabilistic model (DDPM) proposed by Ho et all, which I understand is just a specific ...
- 31
0
votes
0
answers
21
views
KL Divergence in VAE [duplicate]
The basic KL-Divergence between two distributions is as:
$KL(N(\mu_1,\sigma_1) || N(\mu_2, \sigma_2)) = \log \frac{\sigma_2}{\sigma_1} + \frac{\sigma_1^2 + (\mu_1 - \mu_2)^2}{2 \sigma_2^2} - \frac{1}{...
- 225
0
votes
1
answer
78
views
Why does higher dimensional data has higher likelihood?
I am reading about generative models. I came across an example a few times but I cannot come up with an explanation for it.
Imagine data is generated according to $p_\text{data}(x)$. It is often said ...
- 77
0
votes
0
answers
38
views
Why does the reparameterization trick work when some components are still stochastic? [duplicate]
I am trying to understand the reparameterization trick. I got some intuition while looking at this popular question, but I still feel largely confused. I am putting my understanding and doubts here ...
- 500
0
votes
0
answers
21
views
Inference of Beta-Bernoulli Distribution
Assume $x_1, x_2, \cdots, x_n$ follows a $Bern(\pi_0)$, Let $y_{ik}$ follows $Beta(\alpha,\beta)$, $i\in \{1,\cdots, n\}$, and $k\in \{1,\cdots, K\}$. Let $z_k$ follows a Bernoulli Distribution with a ...
- 1
1
vote
0
answers
19
views
Why can Variational Autoencoders (VAEs) approximate arbitrary distributions?
I am trying to reason to myself why is it that VAEs can approximate arbitrary probability distributions even though $q_{\phi}(z|x)$ and $p_{\theta}(x|z)$ are Gaussian.
I understand that the parameters ...
- 21
0
votes
0
answers
22
views
Calculation of an optimal variational distribution for covariance parameters in a Bayesian graphical lasso model
Context:
I am considering here a variational Bayesian framework where I need to calculate the optimal variational distribution for some covariance parameters.
Formally the model can be expressed as:
$$...
- 513
1
vote
0
answers
16
views
Which is the best way to implement variational inference?
To implement variational inference in a Bayesian model, one essentially has the choice between different approaches that differ in their degree of automation and flexibility:
manually deriving update ...
- 11
1
vote
0
answers
24
views
Initialization of parameters of Bayesian Neural Network
What is the best practice to initialize parameters of the Bayesian Neural Network? I am stuck in a situation where I don’t initialize the parameters and every time I run the variational inference, I ...
- 215
1
vote
1
answer
22
views
How can be decoder of VAE represent probability distribution p(x|z) eventhough we directly get image as output
How can be decoder of VAE represent probability distribution p(x|z) eventhough we directly get image as output. Also if my current understanding is right than we get same image for same value of z ...
- 11
0
votes
2
answers
111
views
How to Resolve Variational Autoencoder (VAE) Model Collapse in Reconstruction Task Using Sensor Data?
I am currently experiencing a suspected model collapse in a Variational Autoencoder (VAE) model I am working with. Below are details on the project setup and the issue at hand:
Project Goal: Exploring ...
1
vote
0
answers
11
views
VAE: Noisy decoder output
I'm trying to implement a simple VAE by following several tutorials
like this and this. This is the code that I came up with:
...
- 11
0
votes
0
answers
20
views
How does Variational Autoencoder approximate the joint probability distribution?
I know that in Variational Inference the idea is to approximate the posterior P(z|x, y) and I know that Variational AutoEncoders (VAEs) use the idea of variational inference through neural network ...
- 133
2
votes
1
answer
243
views
Conceptual questions about the proxy distribution in variational inference
I am trying to implement a variational extension of some kind of Bayesian network estimation method. The main goal is to improve speed, since the current method is pretty slow due to MCMC.
My question ...
- 513
1
vote
0
answers
28
views
Prior term in SGHMC implementation
I am working with SGHMC (Stochastic Gradient Hamiltonian Monte Carlo) models.
I found an implimentation of the algorithm in pytorch here. The part of the code that represents momentum variable update (...
2
votes
1
answer
88
views
What is a Student-t VAE ? and how is it different from Gaussian VAE?
I am currently reading https://www.ijcai.org/proceedings/2018/0374.pdf ,this is a research paper based on Student-t Variational Autoencoder for Robust Density Estimation , In this research paper, they ...
- 21
0
votes
0
answers
15
views
Differences between Assumed Density Filtering vs Variational Inference
I am new to the concept of Assumed Density Filtering (ADF) but I think that it is very similar to Variational Inference (VI). In VI, we estimate a intractable posterior distribution (p) by finding a ...
0
votes
0
answers
16
views
Constant terms in variational inference and lower bounds
I have a model for which the lower bound takes the following form
$
\log p(x) \geq \mathbb{E}_{recog}[\log \frac{gen}{recog}] + \mathcal{c}
$
where recog and gen denote the factorization in the ...
- 113
1
vote
0
answers
26
views
Transforming one distribution into another with different support
Consider the following two random variables:
A random vector $S$ with law $h$ and support $\mathcal{S}$ and,
a random vector $X$ with law $c$ and support $\mathcal{X}$.
Assume
$\text{Dim}(\mathcal{...
- 2,386
1
vote
1
answer
81
views
Calculating KL divergence with entropy and cross entropy for VAEs
When looking at implementations of VAE's online, specifically the KL divergence loss, the formula used is:
$$ KL\hspace{1mm} Loss = -\frac{1}{2}(1+\log{\sigma^2}-\mu^2-\sigma^2) $$
or some variation ...
- 23
0
votes
0
answers
15
views
How to put a prior over model parameters?
I am doing a problem in variational inference. I have some data $y$ and I want to understand which distribution it came from. I have the ELBO defined as - $$\text{ELBO}(\phi, D) = \sum_{n=1}^{N} E_{q(...
- 184
1
vote
0
answers
76
views
In the β-TCVAE paper, can someone help with the derivation (S3) in Appendix C.1?
Paper: Isolating Sources of Disentanglement in VAEs
I follow as far as,
$$\mathbb{E}_{q(z)}[log[q(z)] = \mathbb{E}_{q(z, n)}[\ log\ \mathbb{E}_{n'\sim\ p(n)}[q(z|n')]\ ]$$ Subsequently, I don't ...
- 23
0
votes
0
answers
22
views
Graph based variational Autoencoder with variable latent size
I'm trying to build a graph-based Variational-Autoencoder, which should be able to generate graph structures (adjacency matrices). So far, all the papers and models I've seen use a fixed latent vector ...
- 21
0
votes
0
answers
71
views
Handling different image sizes in a VAE
I am working on segmentation and classification of cells based on their shape. After segmentation using CNN my images have different sizes. Next I want to run these images through a VAE for ...
- 1
4
votes
2
answers
274
views
VAE : How is likelihood $p(x|z)$ defined?
Disclaimer : not a strong background in Bayesian statistics.
I gather from questions such as this one and this one that in the context of VAEs, we suppose that we know the (form of the ?) prior $p(z)$ ...
- 1,174
2
votes
2
answers
156
views
Variational inference : is evidence constant?
I'm studying variational inference (in the context of VAEs), and I'm watching this video at this time point. At this point in the video, the goal of approximating the intractable posterior $p_{\theta}(...
- 1,174
3
votes
2
answers
189
views
Is VAE based on mean-field assumption?
According to the slides (section 6), mean-field variational inference (aka. MFVI) assumes the latent variables ($z=\{z_1,..,z_m\}$ are independent from each other, and on top of this assumption, we ...
- 3,429
0
votes
0
answers
299
views
What is the closed-form of the KL-Divergence between two relaxed Bernoulli distributions?
I've seen in multiple papers that use a relaxation of the Bernoulli distribution as defined in Maddison et. al (here it is referred to as Binary Concrete) and they say that a closed form solution for ...
0
votes
1
answer
25
views
How does the decoder predict directly the image in VAEs?
I am reading the VAEs paper Auto-Encoding Variational Bayes. In their loss function:
they define reconstruction loss (second RHS term) as the expected value of the log p(x|z) wrt to the posterior of ...
- 31
0
votes
1
answer
106
views
KL divergence between GPs undefined?
In Theorem 1 in Sun et al, Functional Variational Bayesian Neural Networks, 2019, the authors state the the KL divergence between stochastic processes in the supremum over KL divergence between ...
- 33
1
vote
1
answer
126
views
Mean field approximation convergence
The last sentence of Christopher M. Bishop, Pattern Recognition and Machine Learning Section 10.1.1 Factorized distributions on p.466, states, referring to Equation $(10.9)$, that "Convergence is ...
- 945
0
votes
1
answer
146
views
Deriving the Reparameterization Trick
I am reading about the reparameterization trick from here.
It states
$\boldsymbol{\epsilon}\sim p(\boldsymbol{\epsilon})$, $\textbf{z}=g_\theta(\boldsymbol{\epsilon},\textbf{x})$, and
$$\mathbb{E}_{p_\...
- 143
0
votes
0
answers
34
views
if p(z) and p(x|z) are both gaussian and E(x|z)=z, why does it follow that p(z|x) is also gaussian?
Rocca claims that if we have $p(z)=N(0,1)$ and $p(x|z)=N(z,c)$ where c is a constant, then it would imply that $p(z|x)$ is also gaussian and we can express its mean and variance with respect to the ...
- 3,160
1
vote
0
answers
101
views
Confused about posterior collapse in variational autoencoders
I've been training a $\beta$-VAE with a 5-dimensional latent space on some physics simulation data with 2000 samples. As I increase $\beta$, I notice that an increasing number of the latent variables ...
- 11
1
vote
0
answers
35
views
Genetic Algorithm as engine for Variational Inference?
I'm curious if anyone has used, heard of, or otherwise considered using Genetic Algorithms as an engine for Variational Inference (VI)?
My understanding of VI is that it's an optimization algorithm, ...
- 2,377
1
vote
1
answer
92
views
VAE mean and Standard Deviations are input dependent?
The original presentation of variational autoencoders, VAE assumes the mean $\mu$ and the sd $\sigma$ are functions of the input variable, say $x$. I am studying "Learning Structured Output ...
- 13
2
votes
1
answer
258
views
VQ-VAE uniform prior over the latent variables
VQ-VAE propose to
make the posterior categorical distribution $q(z|x)$ one-hot = defined as $1$ for the closest embedding $e$ in the codebook to the output of the encoder $z_e(x)$;
define a simple ...
- 3,160
0
votes
0
answers
18
views
How does blocked Gibbs Sampling change the interpretation of the generative author-topic (LDA) model
The author topic model is a version of a Latent Dirichlet Allocation model which looks to estimate a set of author to topic, and topic to word distributions to model how authors combine to produce ...
- 11
1
vote
0
answers
119
views
How to maximize the ELBO in coordinate ascent variational inference
In the lecture by D.Blei: https://www.cs.princeton.edu/courses/archive/fall11/cos597C/lectures/variational-inference-i.pdf
Variational inference is explained and he shows how to derive the optimal ...
- 377
0
votes
1
answer
88
views
What should the dimension of the mean and variances vectors in the VAE decoder be?
According to "Autoencoding Variational Bayes" article by Kingma and Welling the decoder part of the VAE should roughly look like this:
...
- 117
4
votes
2
answers
372
views
How does maximizing ELBO in Bayesian neural networks give us the correct posterior predictive distribution?
In Bayesian/variational neural networks one often uses the Evidence Lower BOund (ELBO) as the objective function to optimize with respect to the model parameters. That is if $D=\{y_i,x_i\}_{1\dots n}$ ...
- 121
0
votes
0
answers
19
views
Is it appropriate to discretize conditional posteriors in an MCMC as an alternative to techniques like Metropolis-Hastings or slice-sampling?
Background
Suppose I am interested in sampling the posterior distribution defined by $p(\theta_1,\theta_2|y)$, where $\theta_1,\theta_2$ are parameters of interest and $y$ is a vector of observations. ...
- 391
2
votes
0
answers
28
views
clarification on the local parameterization trick
I was reading Gregory Gundersen's blog entry on the local reparameterization trick
and at a certain point he writes
$$\nabla_{\theta} \mathbb{E}_{p_{\theta}(\bf{z})} [f_{\theta}(\bf{z})]=\int_{\bf{z}} ...
- 325
2
votes
0
answers
202
views
Why is p(x) hard to calculate in a VAE?
In VAEs, it is hard to calculate the posterior: $p_\theta(z|x)$ because that reduces to:
$$
\frac{p(x|z)p(z)}{p(x)}
$$
which is difficult to calculate because $p(x)$ is hard to calculate.
But why ...
- 247
1
vote
1
answer
49
views
Can posterior become tractable if we know p(x)?
In the VAE framework where x is an input data (a vector) and z is a vector of continuous latent variables,
the posterior ...
3
votes
1
answer
648
views
What is the difference between Variational Inference and Variational EM?
I have been reading about variational inference and it is relation to Bayesian regression. It seems there are two versions
The first version is discussed here.
The second version is discussed here.
...
- 288
1
vote
2
answers
604
views
How is the VAE encoder and decoder "probabilistic"?
In a VAE, my understanding is that the encoder takes in $x$, outputs a vector $(\mu, \sigma)$ that characterizes a certain normal distribution $q(z|x)$. Then we sample from this distribution to get a ...
- 173
1
vote
0
answers
395
views
Poor reconstructions from using sigmoid in last layer of variational autoencoder
I have trained a variational autoencoder (VAE) using Pytorch Lightning to reproduce images.
Without sigmoid, reproductions are good. However, some output image ...
- 23
6
votes
3
answers
3k
views
Understanding the Evidence Lower Bound (ELBO)
I am reading this tutorial about Variational Inference, which includes the following depiction of ELBO as the lower bound on log-likelihood on the third page. In the tutorial, $x_i$ is the observed ...
- 1,727