Questions tagged [variational]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
1 vote
0 answers
34 views

Understanding Variational inference and EM in relation to each other

I have read several answers like here but, somehow I still have a few doubts. I hope to present my understanding and ask a few questions to clear my doubts EM: A maximization maximization algorithm E-...
figs_and_nuts's user avatar
0 votes
1 answer
97 views

Understanding a beta-variational autoencoder

I'm working on a beta-variational autoencoder using car images from the Vehicle Color Recognition Dataset. At this point, I'm just exploring different architectures and values for beta. (If you're ...
KirkD_CO's user avatar
  • 1,148
0 votes
0 answers
25 views

Prior estimation in Dynamic (sequence to sequence) Variational Autoencoders (DVAE) with LSTMs

I am trying to implement a sequence-to-sequence variational autoencoder that consists of two parallel sequence encoders. One of the encoders is based on a standard normal prior as in the vanilla vae (...
Nikos H.'s user avatar
0 votes
0 answers
139 views

Confusion with the "lower bound"-term in diffusion models

I am trying to understand the maths of diffusion models following this video explanation on youtube and this blog post. Here is what how I understood it so far: The overall goal is, that we want to ...
mayool's user avatar
  • 1
0 votes
0 answers
216 views

Should the KL loss term for a VAE be the KL-Loss of a batch's mean mu and log sigma, or is it the mean of the kl loss for each individual input image?

I've been trying to learn about Variational Autoencoders and been looking at the Keras sample implementation (https://github.com/keras-team/keras-io/blob/master/examples/generative/vae.py) I'm ...
user1245262's user avatar
0 votes
0 answers
22 views

Understanding line in the derivation of KL divergence optimising function in Variational Bayes

I am following the derivation of Variational Bayes approach in David Blei's lecture notes, particularly equations (13 - 16). In particular, the line: $$ = E_q [\ \log_2 q(Z) ]\ - E_q \left[\ \log_2 \...
Joseph's user avatar
  • 143
3 votes
2 answers
301 views

Replacing the KL-divergence term in a VAE with parameter regularization

When training a VAE, one aim to optimize function $\mathcal{L}$, defined as: $$\mathcal{L}\left(\theta,\phi; \mathbf{x}^{(i)}\right) = - D_{KL}\left(q_\phi(\mathbf{z}|\mathbf{x}^{(i)}) || p_\theta(\...
Asterion's user avatar
  • 936
1 vote
0 answers
93 views

In the β-TCVAE paper, can someone help with the derivation (S3) in Appendix C.1?

Paper: Isolating Sources of Disentanglement in VAEs I follow as far as, $$\mathbb{E}_{q(z)}[log[q(z)] = \mathbb{E}_{q(z, n)}[\ log\ \mathbb{E}_{n'\sim\ p(n)}[q(z|n')]\ ]$$ Subsequently, I don't ...
S R's user avatar
  • 23
4 votes
2 answers
695 views

How does maximizing ELBO in Bayesian neural networks give us the correct posterior predictive distribution?

In Bayesian/variational neural networks one often uses the Evidence Lower BOund (ELBO) as the objective function to optimize with respect to the model parameters. That is if $D=\{y_i,x_i\}_{1\dots n}$ ...
Ian Barnett's user avatar
1 vote
0 answers
115 views

Plot Latent-Space of VAE with embedding or just z_mean?

My model is build on this architecture from github: https://github.com/arogers1/VAE_LSTM_Text_Encoding/blob/master/vae_lstm.py The parameters I found the best results with, included a latent_dim of 16....
NiBa's user avatar
  • 11
3 votes
1 answer
259 views

Does VAE backprop start from the decoder all the way to encoder?

In neural networks that start with input layer, run through hidden layers, and ultimately reach the output layer, we start back-propagation from weights closer to output layer and go backward towards ...
Curious's user avatar
  • 421
1 vote
2 answers
354 views

How disentaglement in latent space can produce poor variety of instances in VAE..?

I'm reading about $\beta$-VAE which essentially proposes a way to disentangle representations in the latent space. We can subjectively (I guess) identify axes carrying specific sources of variations ...
James Arten's user avatar
2 votes
1 answer
453 views

Why do we need Jensen inequality for variational autoencoders?

Just to clarify, I think I understand all the derivations in context of VAEs pretty well; however, there is one last thing that I need explained. There are multiple related derivations of the evidence ...
Tim Joseph's user avatar
4 votes
1 answer
1k views

VAE loss doesn't converge to zero. Does it make sense to sample new instances from trained latent space?

I aim to use a variational autoencoder (VAE) as a generative model. Does this make sense only if the reconstruction loss converges towards zero? On a project I'm working on, the loss is getting ...
James Arten's user avatar
1 vote
0 answers
187 views

Why do we concatenate the condition vector two times in conditional variational autoencoder (CVAE)

I don't quite understand why, in conditional variational autoencoders (CVAEs) we condition on both the encoder and the decoder. In particular, in CVAE the objective function is defined to be: $$\...
James Arten's user avatar
2 votes
0 answers
457 views

How to choose the number of latent dimensions in VAE?

I have trained a VAE that can generate photos of human faces. I have isolated the dimension that correlates most to smiling and now I only want the VAE to generate smiling faces. May I know is it a ...
Johnny Tam's user avatar
0 votes
0 answers
80 views

VAE divergence is positive in minimization of variational inference?

I have been going through the minimization of Variational inference and have a good understanding of all the steps taken: However, there is a part that relies on KL >= 0: I have derived the ...
Frank's user avatar
  • 3
1 vote
1 answer
126 views

Clarification of Equation for Variational Inference in Pattern Recognition and Machine Learning

I am looking at the derivation of variational inference and specifically the approach taken by Bishop in his book on page 465 as illustrated in the Figure below. The key step is the statement below ...
AJR's user avatar
  • 51
4 votes
1 answer
111 views

Rao-Blackwellization in Black Box VI

In the paper, "Black Box Variational Inference," by Ranganath et al. (2013), the authors derive a Rao-Blackwellized estimator of the gradient of the evidence lower bound with respect to a ...
Ethan S's user avatar
  • 41
2 votes
1 answer
2k views

How does the reparameterisation trick work for multivariate Gaussians?

I understand that for sampling from a univariate Gaussian, we can use $x = g(\epsilon) = \mu + \epsilon \sigma $ and then differentiate this transformation with respect to $\mu, \sigma$. How does ...
Ben Gutteridge's user avatar
2 votes
0 answers
59 views

What exactly is the point of computing a lower bound for the log partition function in variational methods in probabilistic graphical models?

Variational methods are applied when we are interested in a probability distribution $P$ but only have a tractably computable unnormalized form $\tilde{P}$ of $P$. Knowing the partition function $Z = \...
user118967's user avatar
1 vote
1 answer
602 views

Generating variations on a class with VQ-VAE with PixelCNN prior

I'm trying to wrap my head around generating from a VQ-VAE with PixelCNN prior. Mostly, I'm curious how to go about generating variations of a given "class", or object. My (foggy) ...
jbm's user avatar
  • 121
3 votes
0 answers
60 views

Using calculus of variations to deduce lower bound on Pitman efficiency (asymptotic relative efficiency) between $t$-test and sign test

Let $Y_1,...,Y_n$ be iid draws from a location family $\{f(\cdot - \theta) : \theta \in \mathbb{R}\}$. $f$ is a symmetric density w.r.t. the Lebesgue measure on $\mathbb{R}$ with finite variance. We ...
martingale_50's user avatar
3 votes
3 answers
2k views

In VAE, why use MSE loss between input x and decoded sample x' from latent distribution?

Variational Autoencoders (VAEs) are based on the concept of Variational Inference (VI) and use two Neural Networks similar to Vanilla Autoencoders (AEs) for function approximation. I understood the ...
Jonas G.'s user avatar
0 votes
0 answers
41 views

Variational Encoder on typical PCA task

I have just started to learn about variational autoencoders. As a first step I have tried doing a task with both PCA and VA. However, the VA results are very poor. Does anybody see any quick fixes ...
KJA's user avatar
  • 111
2 votes
1 answer
1k views

what is -0.5 in VAE loss function with KL term [duplicate]

The VAE loss is composed of two terms: Reconstruction loss KLD loss in the implementation there is -0.5 applied to KLD loss. Kindly let me know what is this -0.5
amir's user avatar
  • 21
2 votes
1 answer
5k views

Mean Square Error as reconstruction loss in VAE

I'm trying to understand one specific formula in a paper that I'm reading: https://arxiv.org/pdf/1911.02469.pdf It's concerning equation 10: Unfortunately, the authors don't explain the context of ...
Sandro's user avatar
  • 125
1 vote
1 answer
30 views

Two-step maximum likelihood inference

Suppose we have an latent r.v. $Z$ (not observed) and an observed r.v. $X$, where $X$ depends on $Z$ via some conditional distribution $p(x|z)$. Given $x$, we will try to infer $z$. Standard maximum ...
D.W.'s user avatar
  • 6,668
2 votes
0 answers
70 views

Proportion of Variance for Variational Autoencoders

For example for PCA, the proportion of variance explained is proportional to the eigenvalue of the respective feature. Now for VAEs, is there a way to estimate the amount of variance that is ...
besterma's user avatar
2 votes
1 answer
141 views

Derivation of the Objective Function for Expectation Propagation

I was reading Expectation Propagation As A Way Of Life and the original paper by Minka Expectation Propagation for Approximate Bayesian Inference and they both say that a fixed point of the EP ...
Euler_Salter's user avatar
  • 2,196
0 votes
1 answer
93 views

Can we test Variational Autoencoders with deterministic $z=0$?

Let's say we want to compare a vanilla Autoencoder to a Variational Autoencoder. The first one gives a deterministic output which basically represents the output with the highest likelihood. When we ...
Sandro's user avatar
  • 125
4 votes
2 answers
543 views

Variational inference with discrete variational parameters

Typically Variational Inference relies on taking gradient steps on KL divergence between the variational and true posterior, or on the ELBO. This does not seem valid when variational parameters are ...
Dion's user avatar
  • 954
2 votes
1 answer
229 views

Variational Inference with intractable score function

Is it possible to do ELBO maximization using stochastic gradient estimates (i.e. iteratively apply variational updates using (3) in http://proceedings.mlr.press/v33/ranganath14.pdf), when it's cheap ...
Dion's user avatar
  • 954
5 votes
1 answer
179 views

Variational inference with deterministic dependencies between variables

Suppose I have a probabilistic graphical model shown in the picture, in which all variables are binary, $c_1$ and $c_2$ are observed, and I want to use mean-field variational inference to estimate ...
Ruben van Bergen's user avatar
4 votes
0 answers
912 views

What's the intuition behind variational learning in Deep NNs with attention mechanism

I'm trying to understand this paper: "Multiple Object Recognition With Visual Attention (Ba et al., 2015)", specifically I'm trying to understand section 3. which explains how the model is trained. ...
Andrew B's user avatar