Stack Exchange Network

Stack Exchange network consists of 174 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Variational Bayesian methods approximate intractable integrals found in Bayesian inference and machine learning. Primarily, these methods serve one of two purposes: Approximating the posterior distribution, or bounding the marginal likelihood of observed data.

0
votes
0answers
6 views

Is the output of a variational autoencoder meant to be a distribution that can be sampled, or a sample directly?

It is difficult to ask this question succinctly in the title, so let me explain. From all the examples of VAEs I have seen, there seem to be 2 approaches used to implement them. In these, ...
0
votes
0answers
14 views

Derivation of ELBO upon the Existence of Conditional Latent Variable Model

I am reading the recently published paper from DeepMind, "Neural Scene Representation and Rendering" and especially its "Supplementary Materials". Following is the page 1 and it's pretty hard for ...
0
votes
0answers
10 views

How to obtain the functional derivative of variational distribution's parameters?

Referring to Collaborative variational autoencoder for recommender systems, I want to know how to derive the parameters for the variational distribution, in the Bayesian inference section of the paper....
1
vote
0answers
9 views

How to obtain the functional derivative of variational distribution?

Referring to , I want to know how to derive the parameters for the variational distribution, in the Bayesian inference section of the paper. I know how to derive, but I don't know how to deduct the ...
1
vote
0answers
13 views

VAE reconstruction vs generation

I have been reading theory on VAE and trying to understand the loss function. The loss function is made up of Reconstruction loss and KL divergence term. What happens when you have good generated ...
0
votes
0answers
26 views

Vae sampling from prior vs posterior

I have been reading the original VAE paper,Auto-Encoding Variational Bayes. In VAE, when generating samples, why do we sample from prior instead of the learned variational posterior(Fig 5 in the paper)...
2
votes
0answers
37 views

Vae reconstruction loss

I have implemeted a Variational Autoencoder(VAE) with a prior different from unit gaussian. This gives me extremely sharp reconstructions compared to normal VAE(small values for reconstruction loss). ...
0
votes
0answers
17 views

Efficient approximate marginal inference in variational auto-encoder

In Auto-Encoding Variational Bayes, authors mentioned that they proposed a solution to "Efficient approximate marginal inference of the variable $x$". I read through the paper and appendix, now ...
0
votes
0answers
19 views

Analytical solution of $D_{KL}(q_\phi (z) || p_\theta (z))$

In appendix of "Auto-Encoding Variational Bayes" by Kingma & Welling, they provide a solution of $D_{KL}(q_\phi (z) || p_\theta (z))$ for Gaussian case. I feel confused about the notation for the ...
3
votes
0answers
28 views

Generative Adversarial Network and Variational Autoencoders for Independent Component Analysis?

Background: I'm working on a model for independent component analysis (ICA) that is based on a methodology similar to GANs and VAEs. What I'm having trouble understanding is how the choice of the loss ...
0
votes
0answers
22 views

Implementing VAE with initial parameter values

I have been trying to implement a Variational Autoencoder. For VAE, we have a unit Gaussian prior and we infer the posterior using the encoder which gives us mean and logvar. I was wondering if ...
0
votes
0answers
37 views

How do we train variational autoencoders in practice?

I learnt that the objective function of a VAE is given by the RHS of the equation $$\ln p(x_n)-KL(q(z|x_n)\Vert p(z|x_n)) = \Bbb E_{z \sim q(z|x_n)}(\ln p(x_n|z))-KL(q(z|x_n) \Vert p(z))$$ in which $...
1
vote
1answer
57 views

Sampling z in VAE

How many times do we sample from $Q(z|x)$ in a Variational Autoencoder? Let’s say that the autoencoder input $x$ is a single image 28x28 pixels - and $Z$ is is a one dimensional distribution. Then, ...
0
votes
1answer
40 views

Variational autoencoder obtain high likelihood but produce low quality sample?

I'm watching Ian Goodfellow's introduction to generative models. When he was introducing variational autoencoders at 22:29, he said: Variational autoencoders are good at obtaining high likelihood,...
2
votes
0answers
34 views

Hierarchical Black Box Variational Inference : Choice of inverse flow

I am reading through Black Box Variational Inference, and having trouble understanding the section for hierarchical inference, where the normalizing flow is introduced. Should this be an arbitrary ...
0
votes
0answers
6 views

Combining AEVB with conjugate exponential family observations

I'm trying to build a probabilistic model that is a combination of a neural network and a graphical model; namely it uses a MLP as an encoder network, and the "decoder" is an exponential family ...
0
votes
0answers
28 views

Diversity of generated samples in VAE

In a variational autoencoder (VAE), it is possible to generate new samples (i.e. images) based on the latent space. After having read quite a few papers about VAE, I still wonder what drives the ...
1
vote
0answers
26 views

How to interpret graphical model for Dirichlet process mixture for variational inference?

I am working through this paper by Blei and Jordan, which introduces variational inference for Dirichlet process mixtures. They derive an evidence lower bound (ELBO) function based on a stick breaking ...
1
vote
0answers
15 views

Why assumptions of exponential family on the complete conditionals imply a conjugacy relationship on other variables

I am reading the Stochastic Variational Inference paper. Basically it first assumes that the joint distribution factorizes into a global term and a product of local terms. $$p(x, z, \beta | \alpha) = ...
2
votes
1answer
55 views

Why in Variational Auto Encoder (Gaussian variational family) we model $\log\sigma^2$ and not $\sigma^2$ (or $\sigma$) itself?

In theory the encoder in VAE (assuming that variational family is Gaussian) generates the $\mu$ and $\sigma$ (or $\sigma^2$). But, in practice, I have seen people assuming the output is $\log\sigma^2$....
5
votes
0answers
193 views

What is the difference between VAE and Stochastic Backpropagation for Deep Generative Models?

What is the difference between Auto-encoding Variational Bayes and Stochastic Backpropagation for Deep Generative Models? Does inference in both methods lead to the same results? I'm not aware of any ...
3
votes
1answer
50 views

Why do Variational Bayes methods assume that the likelihood $p(x|z)$ is tractable while the posterior is not?

I am trying to understand the motivation behind Variational Bayes. I get that the posterior $p(z|x)$ can be intractable, when we would have to compute the evidence with $p(x) = \int p(x|z)p(z) \text{d}...
4
votes
1answer
186 views

Variational autoencoder with Gaussian mixture model

A variational autoencoder (VAE) provides a way of learning the probability distribution $p(x,z)$ relating an input $x$ to its latent representation $z$. In particular, the decoder $d$ maps an input $...
0
votes
1answer
365 views

Loss function autoencoder vs variational-autoencoder or MSE-loss vs binary-cross-entropy-loss

When having real valued entries (e.g. floats between 0 and 1 as normalized representation for greyscale values from 0 to 256) in our label vector, I always thought that we use MSE(R2-loss) if we want ...
2
votes
0answers
36 views

What are the current popular methods for Large Scale Gaussian Process Regression, and which of them are readily available in R?

Vanilla Gaussian Process Regression requires $O(N^3)$ multiplications for estimation, $O(N^2)$ multiplications for prediction and it uses $O(N^2)$ memory where $N$ is the sample size, so it's not ...
0
votes
0answers
38 views

Dependencies for directed graphical models with both probabilistic and deterministic nodes

Based on new developments of variational Bayes recurrent neural networks, I have a question about dependencies over latent variables. I have no problems when there are no deterministic nodes. I can ...
0
votes
0answers
25 views

Help with variational Inference for Hierarchical Dirichlet Process

I am going over this paper by Chong Wang et. al. titled "Online Variational Inference for the Hierarchical Dirichlet Process" and I am struggling with the derivation of equation (17): $\varphi_{jtk} \...
1
vote
2answers
132 views

Variational autoencoder: Why reconstruction term is same to square loss?

In variational autoencoder (see paper), page 5, the loss function for neural networks is defined as: $L(\theta;\phi;x^{i})\backsimeq 0.5*\sum_{j=1}^J(1 + 2\log\sigma^i_j-(\mu^i)^2) - (\sigma^i)^2) + \...
3
votes
1answer
123 views

What can we learn from this visualization?

In here, I learned that the learned MNIST manifold can be visualized as the image below (on page 10, figure 4(b)). My understanding for this visualization is: We start from probability integral ...
1
vote
1answer
42 views

What is a $\propto$ update in mean-field approximation?

I am trying to understand some notation used in papers about Bayesian variational inference. In some papers that use mean-field approximation to fit a probabilistic model, they describe coordinate ...
0
votes
0answers
51 views

Help in Understanding Variational Autoencoders

I get what an autoencoder is. A bottleneck is created by a smaller hidden layer and it acts as nonlinear PCA. But how does a variational autoencoder work? How come it is a generative model? I ...
0
votes
0answers
30 views

Variational Lower Bound in the Bayesian GP-LVM

In the paper [1] we are interested in maximising a lower bound of the form $$ F(q) \geq H(\phi) + \mbox{other terms}, $$ where $H(\phi)$ is given by $$ H(\phi) =\int \phi(u) \left\{ f(u)+\log\frac{p(u)...
0
votes
1answer
72 views

VAE latent vector not taking unit normal distribution

I trained a Convolutional VAE on 1-D electric load curve data (one sample consists of 48-time steps). The training loss (mean square error + KL divergence) decreases during training and is converging. ...
1
vote
1answer
21 views

VAEs: Basic Math example for pass through VAE

As great as many papers and videos are at teaching Neural Net concepts, I see a surprising lack of basic numerical examples explaining these concepts. These examples would let me make sure I know ...
1
vote
0answers
33 views

Jensen's inequality in Collaborative Topic Regression

I am reading the article Collaborative Topic Modeling for Recommending Scientific Articles and could notice the application of Jensen's inequality in order to define a bound from which optimization is ...
0
votes
1answer
201 views

What is the loss function for a probabilistic decoder in the Variational Autoencoder?

In a VAE with Gaussian output the loss function is usually:$$\sum{(\hat x - x)^2} + KL,$$ so the sum of squared errors plus KL divergence. When I also want to predict the variance of the reconstructed ...
1
vote
2answers
86 views

How well does $Q(z|X)$ match $N(0,I)$ in variational autoencoders?

At train time, the KL divergence term drives $Q(z=\mu(X)+\epsilon \times\Sigma(X) | X)$ toward $N(0,I)$, where $\epsilon\sim N(0,I)$. It can't drive $Q(z|X)$ to exactly $N(0,I)$ because the ...
6
votes
1answer
1k views

how to weight KLD loss vs reconstruction loss in variational auto-encoder

in nearly all code examples I've seen of a VAE, the loss functions are defined as follows (this is tensorflow code, but I've seen similar for theano, torch etc. It's also for a convnet, but that's ...
0
votes
2answers
74 views

latent dirichlet allocation: complexity and implementation details

I was confused by how LDA (by the original variational inference) can be implemented in a way such that the number of operations for each document $j$ is $\mathcal{O}(N_j~K)$, where $N_j$ is the ...
1
vote
1answer
59 views

Variational Inference of Univariate Gaussian mixtures

I am reading this paper. In the paper, they use an example of mixture of unit-variance univariate Gaussians with the following parameterization: \begin{align} \mu_k & \sim \mathcal{N}(0, \sigma^2)...
0
votes
0answers
112 views

Comparing ELBO of a VAE for different samples

I am lacking of an interpretation of the evidence lower bound (ELBO), when comparing two different samples $x_1, x_2 \sim X$. Writing the marginal log-likelihood as the sum of lower variational bound ...
1
vote
1answer
97 views

Derivation of Variational Inference

I'm reading Blei et al. (2017) "Variational Inference: A Review for Statisticians" to understand Variational Inference (VI). I follow the paper's notations: $\mathbf{x}_{1:n}$ (observations), $\mathbf{...
1
vote
2answers
103 views

Variational inference: how to rewrite ELBO?

I am reading this paper on variational inference and this website. One thing I am confused about is how they get to decompose ELBO, where $ELBO(q) = E_q[log~p(z,x)] - E_q[log~q(z)]$, when focusing ...
1
vote
1answer
66 views

Computing the gradient of the expectation of a function w.r.t the parameters of the distribution

I am reading this paper on variational auto-encoders by Kingma & Welling and in section 2.2 authors write the following equality for the gradient of the expectation of a function with respect to ...
0
votes
1answer
138 views

Difference between stochastic variational inference and variational inference?

Very simple, as the question header says: what is the difference between SVI and VI? I cannot seem to find a clear-cut definition online.
1
vote
0answers
45 views

Questions about Mean-field variational inference

I am very new to this variational inference concept. I couldn't find any clear sources. I have two questions related to each other. Let's consider a very simple probabilistic model with a 2-D latent ...
1
vote
0answers
325 views

How to write loss function for variational autoencoder?

So I've trying to follow various resources (Geron, Doersch, Altesaar, et al.) to construct a working loss function for my variational autoencoder but I'm finding that formulations either seem to work ...
4
votes
2answers
1k views

Variational Autoencoder VS Normal Autoencoder

I understand the basic structure of variational autoencoder and normal (deterministic) autoencoder and the math behind them, but when and why would I prefer one type of autoencoder to the other? All I ...
0
votes
1answer
62 views

High Dimensional Visualization and VAE Validation

I have a dataset from a black box function, about 35K lines in a text file, with each line containing a single string from the black box function. I am building a VAE to (hopefully) model that data, ...
12
votes
1answer
637 views

What are variational autoencoders and to what learning tasks are they used?

As per this and this answer, autoencoders seem to be a technique that uses neural networks for dimension reduction. I would like to additionally know what is a variational autoencoder (its main ...