Questions tagged [variational-inference]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
0 votes
0 answers
10 views

How to put a prior over model parameters?

I am doing a problem in variational inference. I have some data $y$ and I want to understand which distribution it came from. I have the ELBO defined as - $$\text{ELBO}(\phi, D) = \sum_{n=1}^{N} E_{q(...
0 votes
0 answers
24 views

How to measure posterior collapse if any

Is there any theoretical work on how to measure posterior collapse? One can measure decoder output, but it is not clear if the degradation (if any) happened due to posterior collapse or due to failing ...
1 vote
0 answers
20 views

Are there any methods that combine mcmc and VI?

Are there any methods that combine VI and MCMC? If it exists, why isn’t it used prominently over techniques such as NUTS or other VIs.
  • 11
0 votes
0 answers
10 views

Graph based variational Autoencoder with variable latent size

I'm trying to build a graph-based Variational-Autoencoder, which should be able to generate graph structures (adjacency matrices). So far, all the papers and models I've seen use a fixed latent vector ...
0 votes
0 answers
29 views

Explanation of LDA variational inference derivation

I have a problem deriving one of the variational inference equations for LDA. I am trying to derive the batch variational inference presented in the following paper: https://www.jmlr.org/papers/...
  • 367
2 votes
0 answers
43 views

Why is the Wasserstein distance not used in Variational Inference

I just started learning the concept of variational inference in the context of variational Autoencoder, so please excuse me if the answer is obvious. I would like to know why traditionally, KL-...
0 votes
0 answers
8 views

Recurrent dropout in keras: is the Semeniuta method equivalent to the Gal and Ghahramani method from the point of view of variational inference?

as the title. from this post https://stackoverflow.com/questions/44924690/keras-the-difference-between-lstm-dropout-and-lstm-recurrent-dropout I have recently learned that the implementation of ...
  • 261
2 votes
1 answer
107 views

Justification of the fixed variational distribution in diffusion models

Diffusion models can be regarded as latent variable models (Ho et al., 2020; Section 2), with the latents being an hierarchical chain of random variables $z_T → \dots → z_t → z_{t-1} → \dots → z_1$ (...
4 votes
2 answers
81 views

VAE : How is likelihood $p(x|z)$ defined?

Disclaimer : not a strong background in Bayesian statistics. I gather from questions such as this one and this one that in the context of VAEs, we suppose that we know the (form of the ?) prior $p(z)$ ...
  • 1,074
1 vote
2 answers
62 views

Variational inference : is evidence constant?

I'm studying variational inference (in the context of VAEs), and I'm watching this video at this time point. At this point in the video, the goal of approximating the intractable posterior $p_{\theta}(...
  • 1,074
0 votes
0 answers
24 views

Understanding Variational inference for LDA

I am trying to derive from scratch variational inference for LDA. I am following this course: https://home.cs.colorado.edu/~jbg/teaching/CSCI_5622/19a.pdf When computing $p(Z|\Theta)$ they do the ...
  • 367
0 votes
0 answers
109 views

What is the closed-form of the KL-Divergence between two relaxed Bernoulli distributions?

I've seen in multiple papers that use a relaxation of the Bernoulli distribution as defined in Maddison et. al (here it is referred to as Binary Concrete) and they say that a closed form solution for ...
0 votes
0 answers
7 views

a question on model comparison when 2 different training techniques are used (dropout and variational inference)

i have a doubt : considering 2 different neural networks, one trained through the variational inference technique with denseflipout layers and the other through dropout/concrete dropout In the first ...
  • 261
0 votes
0 answers
44 views

Applying Variational inference to a simple case

My a goal is understand how variational inference works and be able to derive it in a simple way. To do so, I need help deriving variational inference for this simple example. Given a set of labels $L$...
  • 367
0 votes
0 answers
25 views

How to compute an expectation over variational distribution of function values

I am solving a variational inference problem with several variational parameters. One of the variational parameters is a function, where I impose a Gaussian process prior, and I have an associated ...
1 vote
1 answer
89 views

Comparing Gibbs sampler and variational inference

I am learning about variational inference and Gibbs simpler. I am in the process of deriving variational inference on my own. In this process, I need to make a comparison with Gibbs sampler. I am ...
  • 367
3 votes
1 answer
388 views

What's the role of the commitment loss in VQ-VAE?

I'm reading about VQ-VAE, and trying to understand the commitment loss $\beta||z_e(x) - sg(e)||^2$, described in the following sentence: Finally, since the volume of the embedding space is ...
  • 2,820
3 votes
3 answers
81 views

What's the difference between p(Z, X=x) and p(Z|X=x)?

I'm trying to understand variational inference, and I've found resources that mention $p(Z, X=x)$, where $Z$ is a latent random variable and $X$ is the observed random variable. (Here is one such ...
  • 211
0 votes
0 answers
19 views

Variational inference: maximization for sparse gaussian processes

I'm reading the paper on sparse Gaussian processes and I would like to get a proper formalization of the maximization for the lower bound of Equation 27, $$ F_V(X_m, \phi) = \int \phi(\textbf f_m) \...
2 votes
1 answer
214 views

VQ-VAE objective - is it ELBO maximization, or minimization of the KL-divergence between the posterior and its approximation?

I'm reading two descriptions of the VQ-VAE objective: Kingma claims in page 18 that we want to maximize the ELBO, and shows that it can be written as $ELBO = logp_{\theta}(x) - KL(q_{\phi}(z|x)||p_{\...
  • 2,820
1 vote
0 answers
89 views

why is VQ-VAE considered a variational encoder?

I'm reading about VQ-VAE and I'm not sure why do they say we can view it as a VAE. Can you explicitly show: what is the latent z-space - is it the discrete space where z can take the integers 1..K ...
  • 2,820
0 votes
1 answer
180 views

VQ-VAE - why do we need to separate the codebook alignment loss and the commitment loss?

In VQ-VAE, we separate the codebook alignment loss $||sg(z_e(x))-e||^2$ and the commitment loss $||z_e(x)-sg(e)||^2$ where sg stands for the stop-gradient operator, and the loss is $||sg(z_e(x))-e||^2 ...
  • 2,820
4 votes
1 answer
76 views

Why do we approximate the joint in ELBO if we already have access to it?

I realized in variational inference, our goal is to approximate $p(z|x)$ with $q(z)$. So we minimize $KL(q(z) || p(z|x)) = \mathbb{E}_{z \sim q} log\frac{q(z)}{p(z|x)}$. We then manipulate, through ...
  • 211
1 vote
1 answer
45 views

Choice of approximate posterior in variational inference with positive support

I have a simple probabilistic graphical model: $z \longrightarrow x$ where $z_i \sim Exp\left(\lambda_i\right)$ where subscript $i$ denotes the $i$th dimension and $x|z \sim \mathcal{N}\left(f\left(z\...
0 votes
0 answers
28 views

Inverse problem that involves derivatives of unknown function [duplicate]

I am trying to solve an inverse problem where I try to approximate a function $f(x)$ that fulfills the following equation $\ddot{z}_m = \frac{\sigma^2}{2} \frac{\partial^2 f(z_m)}{\partial z^2} + \...
3 votes
1 answer
39 views

Difference between KLdiv(P||Q) and KLdiv(Q||P) in variational inference

Variational inference is about finding an estimation Q(z) for the posterior P(Z|x). According to all the variational inference papers, this is done by minimizing the KLdiv(Q||P). I want to understand ...
  • 367
1 vote
0 answers
25 views

Role of auxiliary objective in semi-supervised VAEs?

In these two papers, mainly: Klys, Jack, Jake Snell, and Richard Zemel. "Learning latent subspaces in variational autoencoders." Advances in neural information processing systems 31 (2018). ...
4 votes
3 answers
1k views

Understanding the Evidence Lower Bound (ELBO)

I am reading this tutorial about Variational Inference, which includes the following depiction of ELBO as the lower bound on log-likelihood on the third page. In the tutorial, $x_i$ is the observed ...
  • 1,677
0 votes
0 answers
33 views

mean-field variational inference in Nonconjugate Models

I'm trying to conduct mean-field Variational Inference (VI) with Nonconjugate model. I found Variational Inference in ...
0 votes
0 answers
22 views

Modifying Variational Inference to be robust to outliers?

Normally, for variational inference, you have some evidence data $Z$, you have some true distribution $P(X|Z)$, and you have a simpler parameterized distribution $Q(X|\theta)$, and you're trying to ...
  • 401
1 vote
2 answers
1k views

VAE: what activation function (if any) to use for the last layer of my decoder if I don't want to assume any knowledge about the scale of my inputs?

I'm working on an implementation of a Variational Autoencoder (VAE). There are lots of helpful examples and guides out there, which typically introduce VAE in the context of image data, e.g. MNIST. ...
  • 133
1 vote
1 answer
33 views

Why do we use the same parameters for the joint, marginal and conditional distributions in VAEs?

I've noticed in several resources on variational autoencoders (for example the Wikipedia article), we use the same parameters theta ($\theta$) for the prior, likelihood, posterior, etc distributions. ...
  • 11
4 votes
1 answer
238 views

Why Reparameterization Trick does not work with discrete latent variables?

I came to know from the Youtube Video here (Timestamp 1:03:55) that Reparameterization trick only works for continuous latent variable. But, I am not clear as to why it does not work for discrete ...
  • 401
4 votes
2 answers
977 views

Prior in variational autoencoders

I am currently dealing with variational autoencoders where I've read the original paper "An introduction to variational Bayes" from Kingma and Welling. I am currently still a little confused ...
  • 41
0 votes
0 answers
76 views

Variational Inference Mean-Field Gaussian

I am new to variational inference and got very confused about some basic ideas. We want to use the mean-field gaussian family to approximate a complicated high-dimensional distribution. I want to ...
  • 87
0 votes
0 answers
31 views

Decoder distribution in avariational autoencoder

I have a little perplexity about the variational autoencoder model, by looking at some underlying terminology. We assume the approximating posterior distribution to be a Gaussian $q_\phi(\textbf{z}|\...
2 votes
2 answers
400 views

Which exact loss do we minimize in a VAE model?

Reading about VAEs here and there, I often get stuck in the confusion about which quantity gets minimized as VAE objective. After some calculations, here's what we get at: $\log p_\theta(x) \ge - \...
0 votes
0 answers
12 views

Constraining reconstructed vectors to lie in a hyperplane ina VAE

I'm trying to add a linear constraint to my variational autoencoder model. Let's say that my input is made of two concatenated vectors: $\textbf{x} = \textbf{t} \oplus \textbf{y}$ where (for example) ...
2 votes
1 answer
183 views

Using a Variational AutoEncoder with an inverse bottleneck

For a problem I'm dealing with, I'm trying to understand if my approach could make sense. I'm using a Variational AutoEncoder (VAE) having relatively low-dimensional inputs, say $x \in \mathbb{R}^n$. ...
0 votes
0 answers
309 views

Higher latent space than Input space in a Variational autoencoder (VAE)

Could it make any sense to choose a larger dimension for the latent space of the VAE with respect to the original input? For example, we may want to learn how to reconstruct a relatively low-...
1 vote
0 answers
197 views

Why is KL divergence used as a measure of closeness in variational inference?

I am curious why KL divergence is the standard measure of (dis)similarity used in VI while it is not even a proper metric (asymmetric and does not satisfy triangle inequality).
  • 273
1 vote
0 answers
33 views

Distribution over parameters vs. distribution over functions

I find it hard to distinguish between these two concepts. In a variational inference setting we learn a distribution over the parameters of our function. in the definition of Gaussian processes we ...
1 vote
1 answer
227 views

Variational Autoencoder assumtions

I am currently reading the paper "Importance Weighted Autoencoders" and am having a hard time understanding something regarding the original Variational Autoencoder (VAE) as described here ...
0 votes
0 answers
35 views

Expected value of log(gamma function(Dirichlet variable))

The following problem emerges from coordinate ascent variational inference in a mixture model with Dirichlet-Multinomial components. I want to compute the expectation of the log likelihood. Since my ...
3 votes
2 answers
144 views

Derivation of ELBO in ADVI Paper, Jacobian of Elliptical Transformation

I've been following the ELBO derivations in the paper Automatic Differentiation Variational Inference and have a few questions. With the model $p(x,\theta)$, they first transform $\theta$ so that it ...
  • 33
1 vote
1 answer
147 views

Variational Bayes for a univariate Gaussian

I'm following an example from Murphy's book (Sec 21.5.1) on how to apply Variational Bayes to infer the posterior over the parameters for a 1D Gaussian $p(\mu,\lambda|\mathcal{D})$. The example uses a ...
  • 605
1 vote
1 answer
72 views

Can someone explain Stein's method/discrepancy in a way that makes sense?

I have been wanting to understand this paper in a deeper way for a long time Stein Variational Gradient Descent: A General Purpose Bayesian Inference Algorithm But everytime I read about Stein's ...
  • 700
0 votes
1 answer
24 views

Stochatic Variational Bayes with distribution on parameters?

In Autoencoding Variational Bayes, the authors show, in the Appendix F, that SGVB can be performed with a model where we have a distribution over the generative parameters \theta (Full VB). They ...
0 votes
0 answers
48 views

Best probability density function to use for the prior of a variance parameter in Bayesian inference

This answer provided some good general advice, but in my specific case I want to create a model of my prior beliefs about the variance of a normally-distributed random variable: $$x \sim \mathcal{N}(0,...
  • 165
2 votes
2 answers
308 views

Inference: How is the Laplace approximation actually useful to us compared with MLE and MAP?

I was reading a few different sources (including the "Machine Learning and Pattern Recognition" book by Bishop) about the Laplace integral approximation method for inference. However, I am ...