Questions tagged [variational-inference]
The variational-inference tag has no usage guidance.
51
questions
0
votes
0
answers
10
views
How to put a prior over model parameters?
I am doing a problem in variational inference. I have some data $y$ and I want to understand which distribution it came from. I have the ELBO defined as - $$\text{ELBO}(\phi, D) = \sum_{n=1}^{N} E_{q(...
- 164
0
votes
0
answers
24
views
How to measure posterior collapse if any
Is there any theoretical work on how to measure posterior collapse?
One can measure decoder output, but it is not clear if the degradation (if any) happened due to posterior collapse or due to failing ...
1
vote
0
answers
20
views
Are there any methods that combine mcmc and VI?
Are there any methods that combine VI and MCMC? If it exists, why isn’t it used prominently over techniques such as NUTS or other VIs.
- 11
0
votes
0
answers
10
views
Graph based variational Autoencoder with variable latent size
I'm trying to build a graph-based Variational-Autoencoder, which should be able to generate graph structures (adjacency matrices). So far, all the papers and models I've seen use a fixed latent vector ...
- 21
0
votes
0
answers
29
views
Explanation of LDA variational inference derivation
I have a problem deriving one of the variational inference equations for LDA.
I am trying to derive the batch variational inference presented in the following paper:
https://www.jmlr.org/papers/...
- 367
2
votes
0
answers
43
views
Why is the Wasserstein distance not used in Variational Inference
I just started learning the concept of variational inference in the context of variational Autoencoder, so please excuse me if the answer is obvious. I would like to know why traditionally, KL-...
- 21
0
votes
0
answers
8
views
Recurrent dropout in keras: is the Semeniuta method equivalent to the Gal and Ghahramani method from the point of view of variational inference?
as the title. from this post https://stackoverflow.com/questions/44924690/keras-the-difference-between-lstm-dropout-and-lstm-recurrent-dropout I have recently learned that the implementation of ...
- 261
2
votes
1
answer
107
views
Justification of the fixed variational distribution in diffusion models
Diffusion models can be regarded as latent variable models (Ho et al., 2020; Section 2), with the latents being an hierarchical chain of random variables $z_T → \dots → z_t → z_{t-1} → \dots → z_1$ (...
- 155
4
votes
2
answers
81
views
VAE : How is likelihood $p(x|z)$ defined?
Disclaimer : not a strong background in Bayesian statistics.
I gather from questions such as this one and this one that in the context of VAEs, we suppose that we know the (form of the ?) prior $p(z)$ ...
- 1,074
1
vote
2
answers
62
views
Variational inference : is evidence constant?
I'm studying variational inference (in the context of VAEs), and I'm watching this video at this time point. At this point in the video, the goal of approximating the intractable posterior $p_{\theta}(...
- 1,074
0
votes
0
answers
24
views
Understanding Variational inference for LDA
I am trying to derive from scratch variational inference for LDA. I am following this course: https://home.cs.colorado.edu/~jbg/teaching/CSCI_5622/19a.pdf
When computing $p(Z|\Theta)$ they do the ...
- 367
0
votes
0
answers
109
views
What is the closed-form of the KL-Divergence between two relaxed Bernoulli distributions?
I've seen in multiple papers that use a relaxation of the Bernoulli distribution as defined in Maddison et. al (here it is referred to as Binary Concrete) and they say that a closed form solution for ...
0
votes
0
answers
7
views
a question on model comparison when 2 different training techniques are used (dropout and variational inference)
i have a doubt :
considering 2 different neural networks, one trained through the variational inference technique with denseflipout layers and the other through dropout/concrete dropout
In the first ...
- 261
0
votes
0
answers
44
views
Applying Variational inference to a simple case
My a goal is understand how variational inference works and be able to derive it in a simple way.
To do so, I need help deriving variational inference for this simple example.
Given a set of labels $L$...
- 367
0
votes
0
answers
25
views
How to compute an expectation over variational distribution of function values
I am solving a variational inference problem with several variational parameters.
One of the variational parameters is a function, where I impose a Gaussian process prior, and I have an associated ...
1
vote
1
answer
89
views
Comparing Gibbs sampler and variational inference
I am learning about variational inference and Gibbs simpler.
I am in the process of deriving variational inference on my own. In this process, I need to make a comparison with Gibbs sampler.
I am ...
- 367
3
votes
1
answer
388
views
What's the role of the commitment loss in VQ-VAE?
I'm reading about VQ-VAE, and trying to understand the commitment loss $\beta||z_e(x) - sg(e)||^2$, described in the following sentence:
Finally, since the volume of the embedding space is ...
- 2,820
3
votes
3
answers
81
views
What's the difference between p(Z, X=x) and p(Z|X=x)?
I'm trying to understand variational inference, and I've found resources that mention $p(Z, X=x)$, where $Z$ is a latent random variable and $X$ is the observed random variable. (Here is one such ...
- 211
0
votes
0
answers
19
views
Variational inference: maximization for sparse gaussian processes
I'm reading the paper on sparse Gaussian processes and I would like to get a proper formalization of the maximization for the lower bound of Equation 27,
$$
F_V(X_m, \phi) = \int \phi(\textbf f_m) \...
2
votes
1
answer
214
views
VQ-VAE objective - is it ELBO maximization, or minimization of the KL-divergence between the posterior and its approximation?
I'm reading two descriptions of the VQ-VAE objective:
Kingma claims in page 18 that we want to maximize the ELBO, and shows that it can be written as $ELBO = logp_{\theta}(x) - KL(q_{\phi}(z|x)||p_{\...
- 2,820
1
vote
0
answers
89
views
why is VQ-VAE considered a variational encoder?
I'm reading about VQ-VAE and I'm not sure why do they say we can view it as a VAE. Can you explicitly show:
what is the latent z-space - is it the discrete space where z can take the integers 1..K ...
- 2,820
0
votes
1
answer
180
views
VQ-VAE - why do we need to separate the codebook alignment loss and the commitment loss?
In VQ-VAE, we separate the codebook alignment loss $||sg(z_e(x))-e||^2$ and the commitment loss $||z_e(x)-sg(e)||^2$ where sg stands for the stop-gradient operator, and the loss is $||sg(z_e(x))-e||^2 ...
- 2,820
4
votes
1
answer
76
views
Why do we approximate the joint in ELBO if we already have access to it?
I realized in variational inference, our goal is to approximate $p(z|x)$ with $q(z)$. So we minimize $KL(q(z) || p(z|x)) = \mathbb{E}_{z \sim q} log\frac{q(z)}{p(z|x)}$. We then manipulate, through ...
- 211
1
vote
1
answer
45
views
Choice of approximate posterior in variational inference with positive support
I have a simple probabilistic graphical model: $z \longrightarrow x$ where $z_i \sim Exp\left(\lambda_i\right)$ where subscript $i$ denotes the $i$th dimension and $x|z \sim \mathcal{N}\left(f\left(z\...
- 141
0
votes
0
answers
28
views
Inverse problem that involves derivatives of unknown function [duplicate]
I am trying to solve an inverse problem where I try to approximate a function $f(x)$ that fulfills the following equation
$\ddot{z}_m = \frac{\sigma^2}{2} \frac{\partial^2 f(z_m)}{\partial z^2} + \...
3
votes
1
answer
39
views
Difference between KLdiv(P||Q) and KLdiv(Q||P) in variational inference
Variational inference is about finding an estimation Q(z) for the posterior P(Z|x).
According to all the variational inference papers, this is done by minimizing the KLdiv(Q||P). I want to understand ...
- 367
1
vote
0
answers
25
views
Role of auxiliary objective in semi-supervised VAEs?
In these two papers, mainly:
Klys, Jack, Jake Snell, and Richard Zemel. "Learning latent subspaces in variational autoencoders." Advances in neural information processing systems 31 (2018).
...
4
votes
3
answers
1k
views
Understanding the Evidence Lower Bound (ELBO)
I am reading this tutorial about Variational Inference, which includes the following depiction of ELBO as the lower bound on log-likelihood on the third page. In the tutorial, $x_i$ is the observed ...
- 1,677
0
votes
0
answers
33
views
mean-field variational inference in Nonconjugate Models
I'm trying to conduct mean-field Variational Inference (VI) with Nonconjugate model.
I found Variational Inference in ...
0
votes
0
answers
22
views
Modifying Variational Inference to be robust to outliers?
Normally, for variational inference, you have some evidence data $Z$, you have some true distribution $P(X|Z)$, and you have a simpler parameterized distribution $Q(X|\theta)$, and you're trying to ...
- 401
1
vote
2
answers
1k
views
VAE: what activation function (if any) to use for the last layer of my decoder if I don't want to assume any knowledge about the scale of my inputs?
I'm working on an implementation of a Variational Autoencoder (VAE). There are lots of helpful examples and guides out there, which typically introduce VAE in the context of image data, e.g. MNIST. ...
- 133
1
vote
1
answer
33
views
Why do we use the same parameters for the joint, marginal and conditional distributions in VAEs?
I've noticed in several resources on variational autoencoders (for example the Wikipedia article), we use the same parameters theta ($\theta$) for the prior, likelihood, posterior, etc distributions. ...
- 11
4
votes
1
answer
238
views
Why Reparameterization Trick does not work with discrete latent variables?
I came to know from the Youtube Video here (Timestamp 1:03:55) that Reparameterization trick only works for continuous latent variable. But, I am not clear as to why it does not work for discrete ...
- 401
4
votes
2
answers
977
views
Prior in variational autoencoders
I am currently dealing with variational autoencoders where I've read the original paper "An introduction to variational Bayes" from Kingma and Welling. I am currently still a little confused ...
- 41
0
votes
0
answers
76
views
Variational Inference Mean-Field Gaussian
I am new to variational inference and got very confused about some basic ideas. We want to use the mean-field gaussian family to approximate a complicated high-dimensional distribution. I want to ...
- 87
0
votes
0
answers
31
views
Decoder distribution in avariational autoencoder
I have a little perplexity about the variational autoencoder model, by looking at some underlying terminology.
We assume the approximating posterior distribution to be a Gaussian $q_\phi(\textbf{z}|\...
- 537
2
votes
2
answers
400
views
Which exact loss do we minimize in a VAE model?
Reading about VAEs here and there, I often get stuck in the confusion about which quantity gets minimized as VAE objective.
After some calculations, here's what we get at:
$\log p_\theta(x) \ge - \...
- 537
0
votes
0
answers
12
views
Constraining reconstructed vectors to lie in a hyperplane ina VAE
I'm trying to add a linear constraint to my variational autoencoder model. Let's say that my input
is made of two concatenated vectors: $\textbf{x} = \textbf{t} \oplus \textbf{y}$ where (for example) ...
- 537
2
votes
1
answer
183
views
Using a Variational AutoEncoder with an inverse bottleneck
For a problem I'm dealing with, I'm trying to understand if my approach could make sense.
I'm using a Variational AutoEncoder (VAE) having relatively low-dimensional inputs, say $x \in \mathbb{R}^n$. ...
- 537
0
votes
0
answers
309
views
Higher latent space than Input space in a Variational autoencoder (VAE)
Could it make any sense to choose a larger dimension for the latent space of the VAE with respect to the original input?
For example, we may want to learn how to reconstruct a relatively low-...
- 537
1
vote
0
answers
197
views
Why is KL divergence used as a measure of closeness in variational inference?
I am curious why KL divergence is the standard measure of (dis)similarity used in VI while it is not even a proper metric (asymmetric and does not satisfy triangle inequality).
- 273
1
vote
0
answers
33
views
Distribution over parameters vs. distribution over functions
I find it hard to distinguish between these two concepts. In a variational inference setting we learn a distribution over the parameters of our function. in the definition of Gaussian processes we ...
- 177
1
vote
1
answer
227
views
Variational Autoencoder assumtions
I am currently reading the paper "Importance Weighted Autoencoders" and am having a hard time understanding something regarding the original Variational Autoencoder (VAE) as described here
...
- 63
0
votes
0
answers
35
views
Expected value of log(gamma function(Dirichlet variable))
The following problem emerges from coordinate ascent variational inference in a mixture model with Dirichlet-Multinomial components. I want to compute the expectation of the log likelihood. Since my ...
- 741
3
votes
2
answers
144
views
Derivation of ELBO in ADVI Paper, Jacobian of Elliptical Transformation
I've been following the ELBO derivations in the paper Automatic Differentiation Variational Inference and have a few questions. With the model $p(x,\theta)$, they first transform $\theta$ so that it ...
- 33
1
vote
1
answer
147
views
Variational Bayes for a univariate Gaussian
I'm following an example from Murphy's book (Sec 21.5.1) on how to apply Variational Bayes to infer the posterior over the parameters for a 1D Gaussian $p(\mu,\lambda|\mathcal{D})$. The example uses a ...
- 605
1
vote
1
answer
72
views
Can someone explain Stein's method/discrepancy in a way that makes sense?
I have been wanting to understand this paper in a deeper way for a long time Stein Variational Gradient Descent: A General
Purpose Bayesian Inference Algorithm
But everytime I read about Stein's ...
- 700
0
votes
1
answer
24
views
Stochatic Variational Bayes with distribution on parameters?
In Autoencoding Variational Bayes, the authors show, in the Appendix F, that SGVB can be performed with a model where we have a distribution over the generative parameters \theta (Full VB). They ...
- 3
0
votes
0
answers
48
views
Best probability density function to use for the prior of a variance parameter in Bayesian inference
This answer provided some good general advice, but in my specific case I want to create a model of my prior beliefs about the variance of a normally-distributed random variable:
$$x \sim \mathcal{N}(0,...
- 165
2
votes
2
answers
308
views
Inference: How is the Laplace approximation actually useful to us compared with MLE and MAP?
I was reading a few different sources (including the "Machine Learning and Pattern Recognition" book by Bishop) about the Laplace integral approximation method for inference. However, I am ...
- 131