Stack Exchange Network

Stack Exchange network consists of 174 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Variational Bayesian methods approximate intractable integrals found in Bayesian inference and machine learning. Primarily, these methods serve one of two purposes: Approximating the posterior distribution, or bounding the marginal likelihood of observed data.

1
vote
1answer
36 views

Using step function as activation function in the final layer

I am using variational autoencoders as machine learning algorithm. My input data are images/matrices that represent user interface layouts or how the HTML page will be divided. I am thinking to ...
0
votes
0answers
17 views

how to calculate loglikelihood for VAE/VQVAE

I asked this question on /r/MLQuestions aswell. Although similar questions have been asked a few times here on reddit and elsewhere, I'm still unclear on how one would calculate the log-likelihood of,...
-1
votes
0answers
5 views

Problem training a (variational) autoencoder on a handwritten signature database [duplicate]

I'm trying to train both a normal autoencoder and a variational autoencoder on a hand written signature database, but it seems like the system is not learning anything, since the reconstructed images ...
2
votes
2answers
53 views

Lower-bound on covariance estimated via Laplace approximation?

I think when a posterior is approximated to be multivariate normal as in Laplace approximation, the covariance matrix is taken to be the negative inverse Hessian evaluated at the log-posterior maximum,...
3
votes
0answers
43 views

Is the optimization of the Gaussian VAE well-posed?

In a Variational Autoencoder (VAE), given some data $x$ and latent variables $t$ with prior distribution $p(t) = \mathcal{N}(t | 0, I)$, the encoder aims to learn a distribution $q_{\phi}(t)$ that ...
0
votes
0answers
32 views

ELBO interpretation in Variational Autoencoder (VAE) for anomaly detection

How to interpret different ELBO values when checking anomaly detection possibilities of VAE model on different "testing" datasets? The higher the ELBO value of the model when testing it on different ...
0
votes
1answer
17 views

Deterministic Decoder for Variational Bayes Autoencoder or RNN

I was wondering if it is possible to use just deterministic function in the output layer of a variational Bayes autoencoder or RNN? Most of papers that I have read are using Gaussian Mixture Models (...
1
vote
1answer
25 views

Do we consider inputs to a VAE to be samples from p(x) and outputs to be samples from p'(x|z) or are they probabilities of x?

In a Variational Autoencoder we say that the encoder and decoder networks model p(z | x) and p(x | z) respectively as seen in the image below: My question is -- is the output of the decoding layer a ...
0
votes
1answer
41 views

Variational Inference: Ising Model

I am self learning Variational Inference. Currently I am reading the chapter 21 book from Murphy 1 and trying to understand the Ising model (21.3.2). The Ising model here is used as denoising ...
0
votes
1answer
119 views

Discrete Random Variables and Deep Generative Models - Why Gumbel-Softmax is needed?

I am reading this 2014 NIPS paper on deep generative models and their application to latent discrete random variables, and this 2017 ICLR paper on Gumbel-Softmax. I essentially don't understand why we ...
1
vote
0answers
28 views

How to choose an appropriate variational distribution?

I work in deep learning research and I am trying to learn how to use variational inference in order to approximate a posterior over the learned weights. I have looked extensively at Yarin Gal's ...
0
votes
0answers
9 views

Using the Expected value of the log as a score for the anomaly detection instead of just the expected value

While dealing with anomaly detection using a probabilistic model I need to compute the probability of an example coming out of the model I built. More specifically: If $p(X)$ is the model I built and ...
0
votes
0answers
35 views

Can the KL Divergence of the ELBO be greater than the log likelihood making it negative?

I have a question about the Evidence Lower Bound or ELBO for variational inference. $$ L = \log p(X)−KL[q(Z)\parallel p(Z\mid X)$$ To my understanding the KL divergence is always $\geq 0.$ However,...
0
votes
1answer
86 views

Defining ELBO in Variational Inference with 3 random variables

I am reading this paper, and having a hard time understanding one of the derivations. It is probably more of a stat question. The context is, having three random variables $x,y,z$, we would want to ...
1
vote
1answer
79 views

If Bayesian approaches are better than frequentist then how can it be as practical?

In a textbook Probability Theory: The Logic of Science written by E. T. Jaynes and others, on page 13 it reads that: For many years, there has been controversy over ‘frequentist’ versus ‘...
0
votes
0answers
17 views

Variational Auto-encoder for supervised learning

It seems that variational auto-encoders (VAE) has become one of the most popular technique for generative modeling. However, is it possible to use variational auto encoders for discriminative ...
0
votes
1answer
48 views

Variational Inference: Computation of ELBO and CAVI algorithm

I am reading/studying this paper 1 and got confused with some expressions. It might be basic for many of you, so my apologizes. In the paper the following prior model is assumed: $\mu_k \sim \...
2
votes
1answer
31 views

When generating samples using variational autoencoder, we decode samples from $N(0,1)$ instead of $\mu + \sigma N(0,1)$

Context: I'm trying to understand the use of variational autoencoders as generators. My understanding: During training, for an input point $x_i$ we want to learn latent $\mu_i$ and $\sigma_i$ and ...
0
votes
2answers
129 views

Reparameterization trick for gamma distribution

I am reading the work of Welling on Vartiational Auto-Encoders (VAE), and wonder if there is any way to generate Gamma distributed samples via a similar reparametrization? The idea of ...
0
votes
1answer
48 views

Is the output of a variational autoencoder meant to be a distribution that can be sampled, or a sample directly?

It is difficult to ask this question succinctly in the title, so let me explain. From all the examples of VAEs I have seen, there seem to be 2 approaches used to implement them. In these, ...
0
votes
0answers
18 views

Derivation of ELBO upon the Existence of Conditional Latent Variable Model

I am reading the recently published paper from DeepMind, "Neural Scene Representation and Rendering" and especially its "Supplementary Materials". Following is the page 1 and it's pretty hard for ...
0
votes
0answers
13 views

How to obtain the functional derivative of variational distribution's parameters?

Referring to Collaborative variational autoencoder for recommender systems, I want to know how to derive the parameters for the variational distribution, in the Bayesian inference section of the paper....
1
vote
0answers
11 views

How to obtain the functional derivative of variational distribution?

Referring to , I want to know how to derive the parameters for the variational distribution, in the Bayesian inference section of the paper. I know how to derive, but I don't know how to deduct the ...
1
vote
0answers
21 views

VAE reconstruction vs generation

I have been reading theory on VAE and trying to understand the loss function. The loss function is made up of Reconstruction loss and KL divergence term. What happens when you have good generated ...
0
votes
0answers
61 views

Vae sampling from prior vs posterior

I have been reading the original VAE paper,Auto-Encoding Variational Bayes. In VAE, when generating samples, why do we sample from prior instead of the learned variational posterior(Fig 5 in the paper)...
2
votes
0answers
118 views

Vae reconstruction loss

I have implemeted a Variational Autoencoder(VAE) with a prior different from unit gaussian. This gives me extremely sharp reconstructions compared to normal VAE(small values for reconstruction loss). ...
0
votes
0answers
22 views

Efficient approximate marginal inference in variational auto-encoder

In Auto-Encoding Variational Bayes, authors mentioned that they proposed a solution to "Efficient approximate marginal inference of the variable $x$". I read through the paper and appendix, now ...
0
votes
0answers
21 views

Analytical solution of $D_{KL}(q_\phi (z) || p_\theta (z))$

In appendix of "Auto-Encoding Variational Bayes" by Kingma & Welling, they provide a solution of $D_{KL}(q_\phi (z) || p_\theta (z))$ for Gaussian case. I feel confused about the notation for the ...
3
votes
0answers
34 views

Generative Adversarial Network and Variational Autoencoders for Independent Component Analysis?

Background: I'm working on a model for independent component analysis (ICA) that is based on a methodology similar to GANs and VAEs. What I'm having trouble understanding is how the choice of the loss ...
0
votes
0answers
26 views

Implementing VAE with initial parameter values

I have been trying to implement a Variational Autoencoder. For VAE, we have a unit Gaussian prior and we infer the posterior using the encoder which gives us mean and logvar. I was wondering if ...
1
vote
0answers
52 views

How do we train variational autoencoders in practice?

I learnt that the objective function of a VAE is given by the RHS of the equation $$\ln p(x_n)-KL(q(z|x_n)\Vert p(z|x_n)) = \Bbb E_{z \sim q(z|x_n)}(\ln p(x_n|z))-KL(q(z|x_n) \Vert p(z))$$ in which $...
1
vote
1answer
138 views

Sampling z in VAE

How many times do we sample from $Q(z|x)$ in a Variational Autoencoder? Let’s say that the autoencoder input $x$ is a single image 28x28 pixels - and $Z$ is is a one dimensional distribution. Then, ...
0
votes
1answer
54 views

Variational autoencoder obtain high likelihood but produce low quality sample?

I'm watching Ian Goodfellow's introduction to generative models. When he was introducing variational autoencoders at 22:29, he said: Variational autoencoders are good at obtaining high likelihood,...
2
votes
0answers
39 views

Hierarchical Black Box Variational Inference : Choice of inverse flow

I am reading through Black Box Variational Inference, and having trouble understanding the section for hierarchical inference, where the normalizing flow is introduced. Should this be an arbitrary ...
0
votes
0answers
7 views

Combining AEVB with conjugate exponential family observations

I'm trying to build a probabilistic model that is a combination of a neural network and a graphical model; namely it uses a MLP as an encoder network, and the "decoder" is an exponential family ...
0
votes
0answers
53 views

Diversity of generated samples in VAE

In a variational autoencoder (VAE), it is possible to generate new samples (i.e. images) based on the latent space. After having read quite a few papers about VAE, I still wonder what drives the ...
1
vote
0answers
27 views

How to interpret graphical model for Dirichlet process mixture for variational inference?

I am working through this paper by Blei and Jordan, which introduces variational inference for Dirichlet process mixtures. They derive an evidence lower bound (ELBO) function based on a stick breaking ...
1
vote
0answers
16 views

Why assumptions of exponential family on the complete conditionals imply a conjugacy relationship on other variables

I am reading the Stochastic Variational Inference paper. Basically it first assumes that the joint distribution factorizes into a global term and a product of local terms. $$p(x, z, \beta | \alpha) = ...
2
votes
1answer
125 views

Why in Variational Auto Encoder (Gaussian variational family) we model $\log\sigma^2$ and not $\sigma^2$ (or $\sigma$) itself?

In theory the encoder in VAE (assuming that variational family is Gaussian) generates the $\mu$ and $\sigma$ (or $\sigma^2$). But, in practice, I have seen people assuming the output is $\log\sigma^2$....
5
votes
0answers
203 views

What is the difference between VAE and Stochastic Backpropagation for Deep Generative Models?

What is the difference between Auto-encoding Variational Bayes and Stochastic Backpropagation for Deep Generative Models? Does inference in both methods lead to the same results? I'm not aware of any ...
3
votes
1answer
52 views

Why do Variational Bayes methods assume that the likelihood $p(x|z)$ is tractable while the posterior is not?

I am trying to understand the motivation behind Variational Bayes. I get that the posterior $p(z|x)$ can be intractable, when we would have to compute the evidence with $p(x) = \int p(x|z)p(z) \text{d}...
4
votes
1answer
445 views

Variational autoencoder with Gaussian mixture model

A variational autoencoder (VAE) provides a way of learning the probability distribution $p(x,z)$ relating an input $x$ to its latent representation $z$. In particular, the decoder $d$ maps an input $...
0
votes
1answer
619 views

Loss function autoencoder vs variational-autoencoder or MSE-loss vs binary-cross-entropy-loss

When having real valued entries (e.g. floats between 0 and 1 as normalized representation for greyscale values from 0 to 256) in our label vector, I always thought that we use MSE(R2-loss) if we want ...
3
votes
0answers
44 views

What are the current popular methods for Large Scale Gaussian Process Regression, and which of them are readily available in R?

Vanilla Gaussian Process Regression requires $O(N^3)$ multiplications for estimation, $O(N^2)$ multiplications for prediction and it uses $O(N^2)$ memory where $N$ is the sample size, so it's not ...
0
votes
0answers
40 views

Dependencies for directed graphical models with both probabilistic and deterministic nodes

Based on new developments of variational Bayes recurrent neural networks, I have a question about dependencies over latent variables. I have no problems when there are no deterministic nodes. I can ...
0
votes
0answers
28 views

Help with variational Inference for Hierarchical Dirichlet Process

I am going over this paper by Chong Wang et. al. titled "Online Variational Inference for the Hierarchical Dirichlet Process" and I am struggling with the derivation of equation (17): $\varphi_{jtk} \...
1
vote
2answers
244 views

Variational autoencoder: Why reconstruction term is same to square loss?

In variational autoencoder (see paper), page 5, the loss function for neural networks is defined as: $L(\theta;\phi;x^{i})\backsimeq 0.5*\sum_{j=1}^J(1 + 2\log\sigma^i_j-(\mu^i)^2) - (\sigma^i)^2) + \...
3
votes
1answer
126 views

What can we learn from this visualization?

In here, I learned that the learned MNIST manifold can be visualized as the image below (on page 10, figure 4(b)). My understanding for this visualization is: We start from probability integral ...
1
vote
1answer
48 views

What is a $\propto$ update in mean-field approximation?

I am trying to understand some notation used in papers about Bayesian variational inference. In some papers that use mean-field approximation to fit a probabilistic model, they describe coordinate ...
0
votes
0answers
59 views

Help in Understanding Variational Autoencoders

I get what an autoencoder is. A bottleneck is created by a smaller hidden layer and it acts as nonlinear PCA. But how does a variational autoencoder work? How come it is a generative model? I ...