Skip to main content

Questions tagged [variational-inference]

Filter by
Sorted by
Tagged with
2 votes
0 answers
37 views

Posterior approximation following optimization methods

I'm trying to quantify the uncertainty in a high dimensional, and multimodal posterior space. We do not have a analytical solution for the forward model, and the forward model could be expensive to ...
Geooo's user avatar
  • 21
0 votes
0 answers
28 views

Variational inference question - how to get eq. 22 from eq. 21

Referring to David Blei's notes on variational inference, I wonder how to get eq. 22 from eq. 21. Also, what is $z_{-k}$ in $L_k = \int q(z_k) \mathbb{E}_{-k}[\log p(z_k | z_{-k},x)]dz_k - \int q(z_k)\...
Mark's user avatar
  • 171
0 votes
0 answers
17 views

Conditions of applications for coordinate ascent variational inference?

In every reference about coordinate ascent variational inference for the mean field family (Chapter 10 Of the book of C.Bishop Pattern recognition and machine learning, or the review article of Blei ...
Pierre Gloaguen's user avatar
1 vote
0 answers
36 views

Using bootstrap for accurate posterior in Variational Bayes

A common well-known issue in Variational Bayes is the variance underestimation of the posterior. Some methods using "sandwich" variance have already been proposed but provide frequentist ...
Mangnier Loïc's user avatar
0 votes
0 answers
37 views

VAE with linear decoder and nonlinear encoder, does this just learn a linear decomposition of the data?

There are a number of variational autoencoder(VAE) methods that have nonlinear encoders and linear decoders. The concept of using the linear decoder is to improve the interpretability (which features ...
sanK's user avatar
  • 1
1 vote
0 answers
35 views

Understanding Variational inference and EM in relation to each other

I have read several answers like here but, somehow I still have a few doubts. I hope to present my understanding and ask a few questions to clear my doubts EM: A maximization maximization algorithm E-...
figs_and_nuts's user avatar
4 votes
3 answers
153 views

Justification of independence assumption for latent variables in Expectation Maximization algorithm

When deriving the ELBO/free energy in the EM algorithm, it is often done in a "general" case of observed and latent variables and then an assumption of independent (or iid) variables is ...
user246795's user avatar
1 vote
1 answer
42 views

How to calculate the score of a new datapoint by a score based diffusion model(song & ermon, 2019)?

I have a pretrained score based diffusion model trained on 64X64 images. Now I want to calculate the score of a new image(of same dimension) through this pre-trained neural network. The score network ...
rajoy99's user avatar
  • 11
1 vote
0 answers
22 views

Using MCMC-derived posterior to design variational approximation function

I am trying to fit a hierarchical model that estimates the covariance of some parameters, using the probabilistic programming language pyro. In simulation experiments, I saw that the MCMC generates ...
David Shor's user avatar
0 votes
0 answers
24 views

On the expressivity of latent variable models

Empirically, we have seen that VAEs can approximate very complex distributions. I am interested in knowing if there are any theoretical results showing how expressive latent variable models can be. ...
Saeed Hedayatian's user avatar
0 votes
0 answers
22 views

Fuzzy rule implementation

I have a rudimentary understanding of fuzzy inference rules of the form IF(x is A) THEN(y is "LOW"). I have not been able to find examples of fuzzy rules that cause a change in the output ...
Brad's user avatar
  • 1
0 votes
0 answers
38 views

Inference of Beta-Bernoulli Distribution

Assume $x_1, x_2, \cdots, x_n$ follows a $Bern(\pi_0)$, Let $y_{ik}$ follows $Beta(\alpha,\beta)$, $i\in \{1,\cdots, n\}$, and $k\in \{1,\cdots, K\}$. Let $z_k$ follows a Bernoulli Distribution with a ...
LAM_MN's user avatar
  • 1
1 vote
0 answers
51 views

Why can Variational Autoencoders (VAEs) approximate arbitrary distributions?

I am trying to reason to myself why is it that VAEs can approximate arbitrary probability distributions even though $q_{\phi}(z|x)$ and $p_{\theta}(x|z)$ are Gaussian. I understand that the parameters ...
Joel's user avatar
  • 21
1 vote
0 answers
27 views

Tree-reweighted belief propagation: optimizing edge appearances $\mu$

I am currently implementing Tree-Reweighted Belief Propagation (TRBP) to optimize edge appearances. The authors in the main manuscript of this work keep the edge appearances, represented by 𝜇, fixed [...
c.uent's user avatar
  • 115
5 votes
1 answer
81 views

Calculation of an optimal variational distribution for covariance parameters in a Bayesian graphical lasso model

Context: I am considering here a variational Bayesian framework where I need to calculate the optimal variational distribution for some covariance parameters. Formally the model can be expressed as: $$...
Mangnier Loïc's user avatar
0 votes
0 answers
25 views

Prior estimation in Dynamic (sequence to sequence) Variational Autoencoders (DVAE) with LSTMs

I am trying to implement a sequence-to-sequence variational autoencoder that consists of two parallel sequence encoders. One of the encoders is based on a standard normal prior as in the vanilla vae (...
Nikos H.'s user avatar
2 votes
1 answer
44 views

Do I need to take additional log det Jacobians for every PDF that uses the reparameterization trick?

Consider the - ELBO objective with reparameterization which is also used in VAE's:$$ \mathcal L_{\theta,\phi}(x)=\log p_\theta(X|Z)+\log p_\theta(Z) +\log q_\phi(Z) $$ The reparameterization trick ...
wd violet's user avatar
  • 777
1 vote
0 answers
43 views

How does Variational Autoencoder approximate the joint probability distribution?

I know that in Variational Inference the idea is to approximate the posterior P(z|x, y) and I know that Variational AutoEncoders (VAEs) use the idea of variational inference through neural network ...
Amir Jalilifard's user avatar
5 votes
1 answer
187 views

derivation of coordinate ascent variational inference

From the slides of variational inference, it shows the evidence lower bound ($L$) and the derivative over a variational distribution $q(z_k)$, quoted as follows $$ L_k = \int q(z_k) E_{-k} \bigg[ \log ...
avocado's user avatar
  • 3,603
3 votes
2 answers
322 views

Replacing the KL-divergence term in a VAE with parameter regularization

When training a VAE, one aim to optimize function $\mathcal{L}$, defined as: $$\mathcal{L}\left(\theta,\phi; \mathbf{x}^{(i)}\right) = - D_{KL}\left(q_\phi(\mathbf{z}|\mathbf{x}^{(i)}) || p_\theta(\...
Asterion's user avatar
  • 936
1 vote
0 answers
98 views

How to measure posterior collapse if any

Is there any theoretical work on how to measure posterior collapse? One can measure decoder output, but it is not clear if the degradation (if any) happened due to posterior collapse or due to failing ...
Pavel Podlipensky's user avatar
1 vote
0 answers
139 views

Are there any methods that combine mcmc and VI?

Are there any methods that combine VI and MCMC? If it exists, why isn’t it used prominently over techniques such as NUTS or other VIs.
JJbox's user avatar
  • 11
2 votes
0 answers
229 views

Why is the Wasserstein distance not used in Variational Inference

I just started learning the concept of variational inference in the context of variational Autoencoder, so please excuse me if the answer is obvious. I would like to know why traditionally, KL-...
user3748950's user avatar
3 votes
1 answer
671 views

Justification of the fixed variational distribution in diffusion models

Diffusion models can be regarded as latent variable models (Ho et al., 2020; Section 2), with the latents being an hierarchical chain of random variables $z_T → \dots → z_t → z_{t-1} → \dots → z_1$ (...
Dan Oneață's user avatar
6 votes
2 answers
913 views

VAE : How is likelihood $p(x|z)$ defined?

Disclaimer : not a strong background in Bayesian statistics. I gather from questions such as this one and this one that in the context of VAEs, we suppose that we know the (form of the ?) prior $p(z)$ ...
Soltius's user avatar
  • 1,316
2 votes
2 answers
329 views

Variational inference : is evidence constant?

I'm studying variational inference (in the context of VAEs), and I'm watching this video at this time point. At this point in the video, the goal of approximating the intractable posterior $p_{\theta}(...
Soltius's user avatar
  • 1,316
0 votes
0 answers
59 views

Understanding Variational inference for LDA

I am trying to derive from scratch variational inference for LDA. I am following this course: https://home.cs.colorado.edu/~jbg/teaching/CSCI_5622/19a.pdf When computing $p(Z|\Theta)$ they do the ...
sam's user avatar
  • 449
0 votes
0 answers
441 views

What is the closed-form of the KL-Divergence between two relaxed Bernoulli distributions?

I've seen in multiple papers that use a relaxation of the Bernoulli distribution as defined in Maddison et. al (here it is referred to as Binary Concrete) and they say that a closed form solution for ...
dannybrig's user avatar
1 vote
1 answer
172 views

Comparing Gibbs sampler and variational inference

I am learning about variational inference and Gibbs simpler. I am in the process of deriving variational inference on my own. In this process, I need to make a comparison with Gibbs sampler. I am ...
sam's user avatar
  • 449
7 votes
1 answer
3k views

What's the role of the commitment loss in VQ-VAE?

I'm reading about VQ-VAE, and trying to understand the commitment loss $\beta||z_e(x) - sg(e)||^2$, described in the following sentence: Finally, since the volume of the embedding space is ...
ihadanny's user avatar
  • 3,340
4 votes
3 answers
108 views

What's the difference between p(Z, X=x) and p(Z|X=x)?

I'm trying to understand variational inference, and I've found resources that mention $p(Z, X=x)$, where $Z$ is a latent random variable and $X$ is the observed random variable. (Here is one such ...
Addison's user avatar
  • 221
3 votes
1 answer
947 views

VQ-VAE objective - is it ELBO maximization, or minimization of the KL-divergence between the posterior and its approximation?

I'm reading two descriptions of the VQ-VAE objective: Kingma claims in page 18 that we want to maximize the ELBO, and shows that it can be written as $ELBO = logp_{\theta}(x) - KL(q_{\phi}(z|x)||p_{\...
ihadanny's user avatar
  • 3,340
2 votes
0 answers
191 views

why is VQ-VAE considered a variational encoder?

I'm reading about VQ-VAE and I'm not sure why do they say we can view it as a VAE. Can you explicitly show: what is the latent z-space - is it the discrete space where z can take the integers 1..K ...
ihadanny's user avatar
  • 3,340
1 vote
1 answer
1k views

VQ-VAE - why do we need to separate the codebook alignment loss and the commitment loss? [duplicate]

In VQ-VAE, we separate the codebook alignment loss $||sg(z_e(x))-e||^2$ and the commitment loss $||z_e(x)-sg(e)||^2$ where sg stands for the stop-gradient operator, and the loss is $||sg(z_e(x))-e||^2 ...
ihadanny's user avatar
  • 3,340
4 votes
1 answer
120 views

Why do we approximate the joint in ELBO if we already have access to it?

I realized in variational inference, our goal is to approximate $p(z|x)$ with $q(z)$. So we minimize $KL(q(z) || p(z|x)) = \mathbb{E}_{z \sim q} log\frac{q(z)}{p(z|x)}$. We then manipulate, through ...
Addison's user avatar
  • 221
1 vote
1 answer
128 views

Choice of approximate posterior in variational inference with positive support

I have a simple probabilistic graphical model: $z \longrightarrow x$ where $z_i \sim Exp\left(\lambda_i\right)$ where subscript $i$ denotes the $i$th dimension and $x|z \sim \mathcal{N}\left(f\left(z\...
isle_of_gods's user avatar
0 votes
0 answers
32 views

Inverse problem that involves derivatives of unknown function [duplicate]

I am trying to solve an inverse problem where I try to approximate a function $f(x)$ that fulfills the following equation $\ddot{z}_m = \frac{\sigma^2}{2} \frac{\partial^2 f(z_m)}{\partial z^2} + \...
can't stop me now's user avatar
3 votes
2 answers
141 views

Difference between KLdiv(P||Q) and KLdiv(Q||P) in variational inference

Variational inference is about finding an estimation Q(z) for the posterior P(Z|x). According to all the variational inference papers, this is done by minimizing the KLdiv(Q||P). I want to understand ...
sam's user avatar
  • 449
1 vote
0 answers
41 views

Role of auxiliary objective in semi-supervised VAEs?

In these two papers, mainly: Klys, Jack, Jake Snell, and Richard Zemel. "Learning latent subspaces in variational autoencoders." Advances in neural information processing systems 31 (2018). ...
MerelyLearning's user avatar
6 votes
3 answers
5k views

Understanding the Evidence Lower Bound (ELBO)

I am reading this tutorial about Variational Inference, which includes the following depiction of ELBO as the lower bound on log-likelihood on the third page. In the tutorial, $x_i$ is the observed ...
nalzok's user avatar
  • 1,787
1 vote
2 answers
3k views

VAE: what activation function (if any) to use for the last layer of my decoder if I don't want to assume any knowledge about the scale of my inputs?

I'm working on an implementation of a Variational Autoencoder (VAE). There are lots of helpful examples and guides out there, which typically introduce VAE in the context of image data, e.g. MNIST. ...
patalt's user avatar
  • 133
1 vote
1 answer
35 views

Why do we use the same parameters for the joint, marginal and conditional distributions in VAEs?

I've noticed in several resources on variational autoencoders (for example the Wikipedia article), we use the same parameters theta ($\theta$) for the prior, likelihood, posterior, etc distributions. ...
Marko's user avatar
  • 11
5 votes
2 answers
542 views

Why Reparameterization Trick does not work with discrete latent variables?

I came to know from the Youtube Video here (Timestamp 1:03:55) that Reparameterization trick only works for continuous latent variable. But, I am not clear as to why it does not work for discrete ...
Curious's user avatar
  • 421
4 votes
2 answers
3k views

Prior in variational autoencoders

I am currently dealing with variational autoencoders where I've read the original paper "An introduction to variational Bayes" from Kingma and Welling. I am currently still a little confused ...
Sidonie's user avatar
  • 41
3 votes
2 answers
1k views

Which exact loss do we minimize in a VAE model?

Reading about VAEs here and there, I often get stuck in the confusion about which quantity gets minimized as VAE objective. After some calculations, here's what we get at: $\log p_\theta(x) \ge - \...
James Arten's user avatar
2 votes
1 answer
404 views

Using a Variational AutoEncoder with an inverse bottleneck

For a problem I'm dealing with, I'm trying to understand if my approach could make sense. I'm using a Variational AutoEncoder (VAE) having relatively low-dimensional inputs, say $x \in \mathbb{R}^n$. ...
James Arten's user avatar
1 vote
0 answers
296 views

Why is KL divergence used as a measure of closeness in variational inference?

I am curious why KL divergence is the standard measure of (dis)similarity used in VI while it is not even a proper metric (asymmetric and does not satisfy triangle inequality).
Sam's user avatar
  • 373
1 vote
0 answers
46 views

Distribution over parameters vs. distribution over functions

I find it hard to distinguish between these two concepts. In a variational inference setting we learn a distribution over the parameters of our function. in the definition of Gaussian processes we ...
samsambakster's user avatar
1 vote
1 answer
542 views

Variational Autoencoder assumtions

I am currently reading the paper "Importance Weighted Autoencoders" and am having a hard time understanding something regarding the original Variational Autoencoder (VAE) as described here ...
Ofek Glick's user avatar
0 votes
0 answers
65 views

Expected value of log(gamma function(Dirichlet variable))

The following problem emerges from coordinate ascent variational inference in a mixture model with Dirichlet-Multinomial components. I want to compute the expectation of the log likelihood. Since my ...
Rylan Schaeffer's user avatar