# Questions tagged [variational-inference]

The tag has no usage guidance.

51 questions
Filter by
Sorted by
Tagged with
10 views

214 views

I'm reading two descriptions of the VQ-VAE objective: Kingma claims in page 18 that we want to maximize the ELBO, and shows that it can be written as ELBO = logp_{\theta}(x) - KL(q_{\phi}(z|x)||p_{\... • 2,820 1 vote 0 answers 89 views ### why is VQ-VAE considered a variational encoder? I'm reading about VQ-VAE and I'm not sure why do they say we can view it as a VAE. Can you explicitly show: what is the latent z-space - is it the discrete space where z can take the integers 1..K ... • 2,820 0 votes 1 answer 180 views ### VQ-VAE - why do we need to separate the codebook alignment loss and the commitment loss? In VQ-VAE, we separate the codebook alignment loss||sg(z_e(x))-e||^2$and the commitment loss$||z_e(x)-sg(e)||^2$where sg stands for the stop-gradient operator, and the loss is$||sg(z_e(x))-e||^2 ...
• 2,820
76 views

I realized in variational inference, our goal is to approximate $p(z|x)$ with $q(z)$. So we minimize $KL(q(z) || p(z|x)) = \mathbb{E}_{z \sim q} log\frac{q(z)}{p(z|x)}$. We then manipulate, through ...
• 211
1 vote
45 views

39 views

### Difference between KLdiv(P||Q) and KLdiv(Q||P) in variational inference

Variational inference is about finding an estimation Q(z) for the posterior P(Z|x). According to all the variational inference papers, this is done by minimizing the KLdiv(Q||P). I want to understand ...
• 367
1 vote
25 views

### Role of auxiliary objective in semi-supervised VAEs?

In these two papers, mainly: Klys, Jack, Jake Snell, and Richard Zemel. "Learning latent subspaces in variational autoencoders." Advances in neural information processing systems 31 (2018). ...
1k views

### Understanding the Evidence Lower Bound (ELBO)

I am reading this tutorial about Variational Inference, which includes the following depiction of ELBO as the lower bound on log-likelihood on the third page. In the tutorial, $x_i$ is the observed ...
• 1,677
33 views

### mean-field variational inference in Nonconjugate Models

I'm trying to conduct mean-field Variational Inference (VI) with Nonconjugate model. I found Variational Inference in ...
22 views

### Modifying Variational Inference to be robust to outliers?

Normally, for variational inference, you have some evidence data $Z$, you have some true distribution $P(X|Z)$, and you have a simpler parameterized distribution $Q(X|\theta)$, and you're trying to ...
• 401
1 vote
1k views

### VAE: what activation function (if any) to use for the last layer of my decoder if I don't want to assume any knowledge about the scale of my inputs?

I'm working on an implementation of a Variational Autoencoder (VAE). There are lots of helpful examples and guides out there, which typically introduce VAE in the context of image data, e.g. MNIST. ...
• 133
1 vote
33 views

### Why do we use the same parameters for the joint, marginal and conditional distributions in VAEs?

I've noticed in several resources on variational autoencoders (for example the Wikipedia article), we use the same parameters theta ($\theta$) for the prior, likelihood, posterior, etc distributions. ...
• 11
238 views

### Why Reparameterization Trick does not work with discrete latent variables?

I came to know from the Youtube Video here (Timestamp 1:03:55) that Reparameterization trick only works for continuous latent variable. But, I am not clear as to why it does not work for discrete ...
• 401
977 views

### Prior in variational autoencoders

I am currently dealing with variational autoencoders where I've read the original paper "An introduction to variational Bayes" from Kingma and Welling. I am currently still a little confused ...
• 41
76 views

### Variational Inference Mean-Field Gaussian

I am new to variational inference and got very confused about some basic ideas. We want to use the mean-field gaussian family to approximate a complicated high-dimensional distribution. I want to ...
• 87
31 views

• 537
12 views

### Constraining reconstructed vectors to lie in a hyperplane ina VAE

I'm trying to add a linear constraint to my variational autoencoder model. Let's say that my input is made of two concatenated vectors: $\textbf{x} = \textbf{t} \oplus \textbf{y}$ where (for example) ...
• 537
183 views

### Using a Variational AutoEncoder with an inverse bottleneck

For a problem I'm dealing with, I'm trying to understand if my approach could make sense. I'm using a Variational AutoEncoder (VAE) having relatively low-dimensional inputs, say $x \in \mathbb{R}^n$. ...
• 537
309 views

### Higher latent space than Input space in a Variational autoencoder (VAE)

Could it make any sense to choose a larger dimension for the latent space of the VAE with respect to the original input? For example, we may want to learn how to reconstruct a relatively low-...
• 537
1 vote
197 views

### Why is KL divergence used as a measure of closeness in variational inference?

I am curious why KL divergence is the standard measure of (dis)similarity used in VI while it is not even a proper metric (asymmetric and does not satisfy triangle inequality).
• 273
1 vote
33 views

### Distribution over parameters vs. distribution over functions

I find it hard to distinguish between these two concepts. In a variational inference setting we learn a distribution over the parameters of our function. in the definition of Gaussian processes we ...
1 vote
227 views

### Variational Autoencoder assumtions

I am currently reading the paper "Importance Weighted Autoencoders" and am having a hard time understanding something regarding the original Variational Autoencoder (VAE) as described here ...
35 views

### Expected value of log(gamma function(Dirichlet variable))

The following problem emerges from coordinate ascent variational inference in a mixture model with Dirichlet-Multinomial components. I want to compute the expectation of the log likelihood. Since my ...
144 views

### Derivation of ELBO in ADVI Paper, Jacobian of Elliptical Transformation

I've been following the ELBO derivations in the paper Automatic Differentiation Variational Inference and have a few questions. With the model $p(x,\theta)$, they first transform $\theta$ so that it ...
• 33
1 vote
147 views

### Variational Bayes for a univariate Gaussian

I'm following an example from Murphy's book (Sec 21.5.1) on how to apply Variational Bayes to infer the posterior over the parameters for a 1D Gaussian $p(\mu,\lambda|\mathcal{D})$. The example uses a ...
• 605
1 vote
72 views

### Can someone explain Stein's method/discrepancy in a way that makes sense?

I have been wanting to understand this paper in a deeper way for a long time Stein Variational Gradient Descent: A General Purpose Bayesian Inference Algorithm But everytime I read about Stein's ...
• 700
24 views

### Stochatic Variational Bayes with distribution on parameters?

In Autoencoding Variational Bayes, the authors show, in the Appendix F, that SGVB can be performed with a model where we have a distribution over the generative parameters \theta (Full VB). They ...
This answer provided some good general advice, but in my specific case I want to create a model of my prior beliefs about the variance of a normally-distributed random variable: x \sim \mathcal{N}(0,...