Questions tagged [variational-inference]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
1 vote
2 answers
49 views

Understanding the Evidence Lower Bound (ELBO)

I am reading this tutorial about Variational Inference, which includes the following depiction of ELBO as the lower bound on log-likelihood on the third page. In the tutorial, $x_i$ is the observed ...
user avatar
  • 1,427
0 votes
0 answers
17 views

mean-field variational inference in Nonconjugate Models

I'm trying to conduct mean-field Variational Inference (VI) with Nonconjugate model. I found Variational Inference in ...
user avatar
0 votes
0 answers
17 views

Modifying Variational Inference to be robust to outliers?

Normally, for variational inference, you have some evidence data $Z$, you have some true distribution $P(X|Z)$, and you have a simpler parameterized distribution $Q(X|\theta)$, and you're trying to ...
user avatar
  • 389
1 vote
2 answers
54 views

VAE: what activation function (if any) to use for the last layer of my decoder if I don't want to assume any knowledge about the scale of my inputs?

I'm working on an implementation of a Variational Autoencoder (VAE). There are lots of helpful examples and guides out there, which typically introduce VAE in the context of image data, e.g. MNIST. ...
user avatar
  • 133
1 vote
1 answer
23 views

Why do we use the same parameters for the joint, marginal and conditional distributions in VAEs?

I've noticed in several resources on variational autoencoders (for example the Wikipedia article), we use the same parameters theta ($\theta$) for the prior, likelihood, posterior, etc distributions. ...
user avatar
  • 11
3 votes
1 answer
78 views

Why Reparameterization Trick does not work with discrete latent variables?

I came to know from the Youtube Video here (Timestamp 1:03:55) that Reparameterization trick only works for continuous latent variable. But, I am not clear as to why it does not work for discrete ...
user avatar
  • 161
2 votes
2 answers
57 views

Prior in variational autoencoders

I am currently dealing with variational autoencoders where I've read the original paper "An introduction to variational Bayes" from Kingma and Welling. I am currently still a little confused ...
user avatar
  • 21
0 votes
0 answers
15 views

Variational Inference Mean-Field Gaussian

I am new to variational inference and got very confused about some basic ideas. We want to use the mean-field gaussian family to approximate a complicated high-dimensional distribution. I want to ...
user avatar
  • 87
0 votes
0 answers
19 views

Decoder distribution in avariational autoencoder

I have a little perplexity about the variational autoencoder model, by looking at some underlying terminology. We assume the approximating posterior distribution to be a Gaussian $q_\phi(\textbf{z}|\...
user avatar
2 votes
1 answer
24 views

Which exact loss do we minimize in a VAE model?

Reading about VAEs here and there, I often get stuck in the confusion about which quantity gets minimized as VAE objective. After some calculations, here's what we get at: $\log p_\theta(x) \ge - \...
user avatar
0 votes
0 answers
9 views

Constraining reconstructed vectors to lie in a hyperplane ina VAE

I'm trying to add a linear constraint to my variational autoencoder model. Let's say that my input is made of two concatenated vectors: $\textbf{x} = \textbf{t} \oplus \textbf{y}$ where (for example) ...
user avatar
1 vote
1 answer
46 views

Using a Variational AutoEncoder with an inverse bottleneck

For a problem I'm dealing with, I'm trying to understand if my approach could make sense. I'm using a Variational AutoEncoder (VAE) having relatively low-dimensional inputs, say $x \in \mathbb{R}^n$. ...
user avatar
0 votes
0 answers
93 views

Higher latent space than Input space in a Variational autoencoder (VAE)

Could it make any sense to choose a larger dimension for the latent space of the VAE with respect to the original input? For example, we may want to learn how to reconstruct a relatively low-...
user avatar
0 votes
0 answers
37 views

Why is KL divergence used as a measure of closeness in variational inference?

I am curious why KL divergence is the standard measure of (dis)similarity used in VI while it is not even a proper metric (asymmetric and does not satisfy triangle inequality).
user avatar
  • 223
0 votes
0 answers
26 views

Appendix A on Variational Gaussian Process State Space Model

On Frigola et al in the Supplementary material A, equation (19) is: $\prod_{t=1}^{T}p(\mathbf{f}_t|\mathbf{f}_{1:t-1},\mathbf{x}_{0:t-1},\mathbf{u})=\mathcal{N}(\mathbf{f}_{1:T}|\mathbf{K}_{0:T-1,\...
user avatar
0 votes
0 answers
23 views

Distribution over parameters vs. distribution over functions

I find it hard to distinguish between these two concepts. In a variational inference setting we learn a distribution over the parameters of our function. in the definition of Gaussian processes we ...
user avatar
1 vote
1 answer
56 views

Variational Autoencoder assumtions

I am currently reading the paper "Importance Weighted Autoencoders" and am having a hard time understanding something regarding the original Variational Autoencoder (VAE) as described here ...
user avatar
0 votes
0 answers
25 views

Expected value of log(gamma function(Dirichlet variable))

The following problem emerges from coordinate ascent variational inference in a mixture model with Dirichlet-Multinomial components. I want to compute the expectation of the log likelihood. Since my ...
user avatar
0 votes
0 answers
10 views

Bayesian Interpretation of Deep Ensembles

I was wondering if training a neural network in the deep ensemble setting can lead to a network with a posterior vs. a point estimate architecture? Recently there have been discussions over the ...
user avatar
0 votes
0 answers
24 views

Best method(s) to estimate the parameters of a stochastic process with a hybrid (i.e. switching) random input variable?

I'm looking for the best approach to and/or methods of solving the following inference problem. I have tried searching for similar questions but don't have enough knowledge on the various methods (...
user avatar
  • 155
3 votes
2 answers
80 views

Derivation of ELBO in ADVI Paper, Jacobian of Elliptical Transformation

I've been following the ELBO derivations in the paper Automatic Differentiation Variational Inference and have a few questions. With the model $p(x,\theta)$, they first transform $\theta$ so that it ...
user avatar
  • 33
0 votes
0 answers
70 views

Variational inference - posterior predictive distribution

So suppose we have a neural network that aims to map values from one distribution to another. That is to say the inputs do not belong to the same distribution as the targets. It then follows that, ...
user avatar
1 vote
1 answer
99 views

Variational Bayes for a univariate Gaussian

I'm following an example from Murphy's book (Sec 21.5.1) on how to apply Variational Bayes to infer the posterior over the parameters for a 1D Gaussian $p(\mu,\lambda|\mathcal{D})$. The example uses a ...
user avatar
  • 643
0 votes
0 answers
16 views

Inhowfar does Variational Inference work better with members of the exponential family?

I am reading Variational Inference: A Review for Statisticians. Working in [the exponential] family simplifies variational inference: it is easier to derive the corresponding CAVI algorithm, and it ...
user avatar
0 votes
0 answers
37 views

Variational inference: Rewriting ELBO

I was following some tutorials on variational inference and ELBO loss functions and I think I understand it quite well, but I'm struggling with math. If I understand it correctly we went from this ...
user avatar
  • 1
0 votes
0 answers
98 views

Loss Function of a Variational Autoencoder when using Implicit Reparameterization Gradients (Dirichlet distributed latent space)

I would like to implement a VAE with a Dirichlet distributed latent space in Python. Since the reparametrization trick does not work for the Dirichlet Distribution I would use Implicit ...
user avatar
0 votes
0 answers
89 views

Variational Autoencoder with Dirichlet distributed latent space using the Weibull Distribution

My goal is to create an VAE with an Dirichlet distributed latent space. Since the reparametrization trick does not work for the Dirichlet Distribution, I am trying to approximate the Gamma ...
user avatar
0 votes
1 answer
38 views

Can someone explain Stein's method/discrepancy in a way that makes sense?

I have been wanting to understand this paper in a deeper way for a long time Stein Variational Gradient Descent: A General Purpose Bayesian Inference Algorithm But everytime I read about Stein's ...
user avatar
  • 629
0 votes
0 answers
43 views

Variational coin tossing from scratch: calculation of the expected log likelihood

I'm working my way through this tutorial about variational inference for a coin tossing. Let's say the probability of the event head is denoted by ...
user avatar
0 votes
1 answer
20 views

Stochatic Variational Bayes with distribution on parameters?

In Autoencoding Variational Bayes, the authors show, in the Appendix F, that SGVB can be performed with a model where we have a distribution over the generative parameters \theta (Full VB). They ...
user avatar
0 votes
0 answers
19 views

Derivation of updating rule

I am trying to understand "Variational Bayesian Multiple Instance Learning with Gaussian Processes" by Manuel Haußmann, Fred A. Hamprecht, and Melih Kandemir. Right now I am stuck in the ...
user avatar
0 votes
0 answers
32 views

Best probability density function to use for the prior of a variance parameter in Bayesian inference

This answer provided some good general advice, but in my specific case I want to create a model of my prior beliefs about the variance of a normally-distributed random variable: $$x \sim \mathcal{N}(0,...
user avatar
  • 155
2 votes
2 answers
146 views

Inference: How is the Laplace approximation actually useful to us compared with MLE and MAP?

I was reading a few different sources (including the "Machine Learning and Pattern Recognition" book by Bishop) about the Laplace integral approximation method for inference. However, I am ...
user avatar
0 votes
0 answers
21 views

Differentiating entropy in Reinforcement Learning as Probabilistic Inference

I am studying the paper Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Review (https://arxiv.org/abs/1805.00909) and I do not understand how the author differentiate the ...
user avatar
0 votes
1 answer
99 views

variational parameters in variational autoencoders

So we have x as the observed variable, and z as the latent variable, denoted by this bayesian network. And we parameterize x by $\theta$ to get $p_{\theta}(\mathbf{x})$ the posterior is $p_{\theta}(...
user avatar
  • 103
0 votes
0 answers
55 views

Areas of research in statistical machine learning

I'm reading from a book called Machine Learning: A Probabilistic Perspective by Kevin Murphy. Besides being somewhat challenging to understand, I feel that the earlier chapters (on probability, ...
0 votes
0 answers
14 views

Understanding variational inference for Bayesian GMM

I am reading Variational Inference: A Review for Statisticians by Blei, Kucukelbir and McAuliffe. I am having a hard time following some of the steps in Section 3.2 More precisely, they state that $\...
user avatar
  • 793