Questions tagged [variational-bayes]

Variational Bayesian methods approximate intractable integrals found in Bayesian inference and machine learning. Primarily, these methods serve one of two purposes: Approximating the posterior distribution, or bounding the marginal likelihood of observed data.

Filter by
Sorted by
Tagged with
1 vote
0 answers
47 views

Poor reconstructions from using sigmoid in last layer of variational autoencoder

I have trained a variational autoencoder (VAE) using Pytorch Lightning to reproduce images. Without sigmoid, reproductions are good. However, some output image ...
user avatar
  • 23
0 votes
0 answers
32 views

Posterior predictive distribution: Sampling vs calculating

I am having trouble understanding how to make predictions with the posterior predictive distribution. Posterior predictive is $p(y|x,D)=\int p(y|x,\theta)p(\theta|D) d\theta$ where $D$ is the training ...
user avatar
  • 371
1 vote
2 answers
41 views

Understanding the Evidence Lower Bound (ELBO)

I am reading this tutorial about Variational Inference, which includes the following depiction of ELBO as the lower bound on log-likelihood on the third page. In the tutorial, $x_i$ is the observed ...
user avatar
  • 1,427
0 votes
0 answers
17 views

Modifying Variational Inference to be robust to outliers?

Normally, for variational inference, you have some evidence data $Z$, you have some true distribution $P(X|Z)$, and you have a simpler parameterized distribution $Q(X|\theta)$, and you're trying to ...
user avatar
  • 389
1 vote
0 answers
23 views

Variational autoencoder for varying-length matrix-valued data

I have a data set $\mathcal D \subset \mathcal X$ where each observation is itself a mini dataset, each with the same number of columns but differing numbers of rows, so $\mathcal D = \{X_1, \dots, ...
user avatar
  • 601
1 vote
2 answers
50 views

VAE: what activation function (if any) to use for the last layer of my decoder if I don't want to assume any knowledge about the scale of my inputs?

I'm working on an implementation of a Variational Autoencoder (VAE). There are lots of helpful examples and guides out there, which typically introduce VAE in the context of image data, e.g. MNIST. ...
user avatar
  • 133
1 vote
0 answers
60 views

Classical VAE not learning 2D gaussian mixture distribution using MSE loss

I've been exploring VAE for non-image data. I consider small to medium-sized continuous vector spaces and I want to learn the distribution of a dataset in that space. As a warm up exercise, I tried ...
user avatar
  • 113
1 vote
1 answer
23 views

Why do we use the same parameters for the joint, marginal and conditional distributions in VAEs?

I've noticed in several resources on variational autoencoders (for example the Wikipedia article), we use the same parameters theta ($\theta$) for the prior, likelihood, posterior, etc distributions. ...
user avatar
  • 11
0 votes
0 answers
18 views

Question regarding the Bayes estimator of a parameter $\theta$ which has a continuous distribution

Let $c > 0$ and \begin{equation} L(\theta,a)=\left\{ \begin{array}{@{}ll@{}} c|\theta-a|, & \text{if}\ \theta < a \\ |\theta-a|, & \text{if}\ \theta \ge a \quad. \end{array}\...
user avatar
2 votes
2 answers
54 views

Prior in variational autoencoders

I am currently dealing with variational autoencoders where I've read the original paper "An introduction to variational Bayes" from Kingma and Welling. I am currently still a little confused ...
user avatar
  • 21
1 vote
0 answers
53 views

Simplifying the Kullback-Leibler divergence for a sum of distributions

I want to find an approximation of a mixture of probability distributions that minimises the Kullback-Leibler divergence (KLD). I need to verify my result, as it seems suspect. We have a joint ...
user avatar
  • 161
1 vote
1 answer
29 views

Which parameters are updated in VAE with normalizing flow?

I've been reading this article about implementing a VAE with normalizing flows. What it's not clear to me, is which parameters are actually optimized using this approach. Should I only optimize the ...
user avatar
1 vote
0 answers
54 views

formulation of evidence lower bound (ELBO) of the log likelihood

Since the evidence lower bound (ELBO) of the log likelihood, $\log p_{\theta}(x)$, is $$\mathbb{E}_{q_\phi(z|x)} [\log p_\theta(x|z)] - \text{KL}(q_\phi(z|x), p(z)).$$But we can also write the above ...
user avatar
  • 111
0 votes
0 answers
29 views

How to evaluate quality of VAEs generated samples

I have a set of generated samples from a latent distribution (say 100 images) from a learned VAE. For GANs, the Inception score metric (which helps assess image quality and image diversity). Any idea ...
user avatar
  • 1
0 votes
0 answers
31 views

Help understanding variational autoencoder with learned latent structure

I’m reading and trying to understand the Variational Autoencoder with Learned latent structure (https://arxiv.org/abs/2006.10597). My understandings are they use the transport operators to define the ...
user avatar
  • 1
1 vote
1 answer
55 views

Variational Autoencoder assumtions

I am currently reading the paper "Importance Weighted Autoencoders" and am having a hard time understanding something regarding the original Variational Autoencoder (VAE) as described here ...
user avatar
1 vote
1 answer
78 views

Generative model that satisfies certain algebraic constraints

Disclaimer: I need guidance and help with where to start looking for solution of the problem I have described below. My background is in optimization and I am new to statistical methods, so there is a ...
user avatar
1 vote
1 answer
46 views

Clarification on normal distribution assumption for probablistic decoder P(x|z) in VAE

After searching around on the internet, it seems that the following assumption seems to be commonly used in VAEs: For a continuous domain (MNIST), assume $p(x|z) = N(x; f(z), \sigma^2)$. Where $f(z)$ ...
user avatar
0 votes
0 answers
67 views

Variational inference - posterior predictive distribution

So suppose we have a neural network that aims to map values from one distribution to another. That is to say the inputs do not belong to the same distribution as the targets. It then follows that, ...
user avatar
1 vote
1 answer
98 views

Variational Bayes for a univariate Gaussian

I'm following an example from Murphy's book (Sec 21.5.1) on how to apply Variational Bayes to infer the posterior over the parameters for a 1D Gaussian $p(\mu,\lambda|\mathcal{D})$. The example uses a ...
user avatar
  • 631
0 votes
0 answers
37 views

Variational inference: Rewriting ELBO

I was following some tutorials on variational inference and ELBO loss functions and I think I understand it quite well, but I'm struggling with math. If I understand it correctly we went from this ...
user avatar
  • 1
0 votes
0 answers
75 views

Is multicollinearity a problem when fitting a Bayesian regression model using ADVI?

If I’m fitting a bayesian regression model using ADVI, is it important to ensure all the covariates are uncorrelated with each other? I have a vague understanding that ADVI doesn’t play well with ...
user avatar
  • 29
0 votes
0 answers
19 views

Evaluating evidence lower bound for Poisson factorisation model

tl;dr This question is about evaluating the evidence lower bound (ELBO) for variational inference in the context of count data. Background I am trying to reproduce hierarchical Poisson factorisation ...
user avatar
1 vote
1 answer
48 views

forward and reverse KL divergence for variational inference

I have a question regarding the forward or reverse KL divergence used in variational inference. In accordance with the following lecture notes, reverse KL can cause q under-estimate the support of p ...
user avatar
  • 2,637
0 votes
1 answer
20 views

Stochatic Variational Bayes with distribution on parameters?

In Autoencoding Variational Bayes, the authors show, in the Appendix F, that SGVB can be performed with a model where we have a distribution over the generative parameters \theta (Full VB). They ...
user avatar
0 votes
1 answer
43 views

Marginal likelihood: Why is it difficult to compute in this case?

I have been reading up a bit on generative models particularly trying to understand the math behind VAE. While looking at a talk online, the speaker mentions the following definition of marginal ...
user avatar
  • 4,430
0 votes
0 answers
50 views

How to optimize a function with respect to a distribution, in the context of variational inference

Context: I am learning about variational inference. The reference I am following is linked at the end of this post. Goal: I want to learn how a marginal variational distribution $q_k$ is optimal in ...
user avatar
  • 2,101
0 votes
0 answers
27 views

Reference Request: Variational Expectation-Maximization algorithm for Latent Dirichlet Allocation with an added time component

This link has a pretty good runthrough on the variational inference (via variational E-M) for LDA with calculations expanded and explained. I am now considering a modified LDA which adds a time ...
user avatar
  • 1
2 votes
2 answers
146 views

Inference: How is the Laplace approximation actually useful to us compared with MLE and MAP?

I was reading a few different sources (including the "Machine Learning and Pattern Recognition" book by Bishop) about the Laplace integral approximation method for inference. However, I am ...
user avatar
0 votes
0 answers
24 views

Algorithm of variational autoencoders

Recently I was reading about variational autoencoders and I'm not sure if I correctly understand the algorithm of it. I also met very good lecture on you tube about autoencoders and there I found the ...
user avatar
  • 163
2 votes
1 answer
106 views

Semi-supervised classification objective from Kingma et al

In this 2014 paper, Kingma et al. develop different methods to do semi-supervised learning with VAEs. In one of their proposed solutions ("M2"), they approach this problem by incorporating ...
user avatar
2 votes
0 answers
30 views

How to derive local variational close form for logistic regression, analytically

In Prof Bishop's book, Pattern Recognition and Machine Learning, Chapter 10, on local variational inference for logistic regression, page 501, equation 10.161 has been derived, from differentiating EM ...
user avatar
  • 143
0 votes
0 answers
13 views

Understanding variational inference for Bayesian GMM

I am reading Variational Inference: A Review for Statisticians by Blei, Kucukelbir and McAuliffe. I am having a hard time following some of the steps in Section 3.2 More precisely, they state that $\...
user avatar
  • 793
0 votes
1 answer
51 views

Regression in Baysian settings

Assume we have the posterior distribution of this linear regression model $y = w^Tx$, $P(w | D,\theta)$, where $D = \{(x_i,y_i)\}_{i \in \{1,\dots,n\}}, n $ is the number of data instances, $\theta$ ...
user avatar
  • 231
0 votes
0 answers
51 views

Can a bayesian neural network with independent normal variational distributions over weights and biases produce a multi-modal posterior?

For a project I am involved in, I am doing surrogate modelling. This means that I simulate data $\mathcal{D}=\{X,Y\}$ that is used to train some probabilistic non-linear regression model. The model is ...
user avatar
1 vote
0 answers
24 views

The Role Variational Free Energy or Expected Lower Bound (ELBO) Plays as a Loss Function

In reference to variational free energy or expected lower bound, I found this sentence, "As one can easily see, the cost function tries to balance the complexity of the data P(D | w) and the ...
user avatar
0 votes
2 answers
80 views

In bayesian approach, what is the difference between full posterior and MAP [duplicate]

Consider a classic machine learning problem, which we want to solve using NN. And suppose that we want to use bayesian learning for that. In the bayesian approach the posterior is described as follows:...
user avatar
  • 1,033
1 vote
1 answer
87 views

Clarification of Equation for Variational Inference in Pattern Recognition and Machine Learning

I am looking at the derivation of variational inference and specifically the approach taken by Bishop in his book on page 465 as illustrated in the Figure below. The key step is the statement below ...
user avatar
  • 51
0 votes
1 answer
45 views

Is outer product of marginal distribution the "best" mean-field approximation for a joint distribution?

I am certain this has been asked somewhere else, if that's the case, link me and close the thread. I am studying variational inference and mean-field approximation. All the explanations I come across ...
user avatar
  • 103
2 votes
1 answer
76 views

In VAE, why are we approximating p(z|x) using q(z) and not q(z|x) [duplicate]

I am watching this lecture on VAE: https://www.youtube.com/watch?v=uaaqyVS9-rM&t=1507s and at 26:00, it is stated that the goal is the minimize the KL div. between the distribution we are trying ...
user avatar
1 vote
1 answer
31 views

Approximating the posterior and learning the distribution over the weights after training

I am familiar with the methods in variational inference in which after training we have access to the distribution over the network's weights. This is necessary for estimating epistemic uncertainty. ...
user avatar
3 votes
1 answer
193 views

How do we get to the MSE in the loss function for a variational autoencoder?

Context: https://arxiv.org/pdf/1312.6114.pdf So if I start with this equation: $$ \mathcal{L}\left(\boldsymbol{\theta}, \boldsymbol{\phi} ; \mathbf{x}^{(i)}\right) \simeq \frac{1}{2} \sum_{j=1}^{J}\...
user avatar
  • 93
0 votes
0 answers
91 views

Convergence of CAVI(Coordinate Ascent Variational Inference)

I was reading several resources on variational inference, and most of them stated that the CAVI algorithm converges to local maximum, and Bishop's textbook stated that the convergence is guaranteed as ...
user avatar
1 vote
0 answers
58 views

I am confused about the sum indices of these posterior distribution formulas

I am reading this notes and trying to understand how the posterior distribution formulas of the variational variables involved have been calculated. I am confused about the indices of the summation. ...
user avatar
1 vote
0 answers
26 views

Tied Bayesian Mixture of Gaussians

I am bit confused when it comes to modelling a Bayesian Gaussian mixture model that assumes a shared covariance/precision matrix for all Gaussian components. I followed the derivation in Bishop and ...
user avatar
  • 11
1 vote
1 answer
232 views

Why do we start from $KLD(q(z) || p(z|x))$ in ELBO derivation?

Of the several ways to derive the evidence lower bound (such as using Jensen’s inequality), a version often used is the derivation from $KLD(q(z) || p(z|x))$. The following image illustrates the ...
user avatar
1 vote
0 answers
89 views

Is there a reason to use variational inference for point estimates?

I have seen Bayesian hierarchical models, particularly in computational biology, that use variational inference, but do not use the uncertainty provided by a variational solution. For example, MOFA is ...
user avatar
0 votes
3 answers
321 views

Why Assumed Density Filtering is also called Moment Matching?

I am learning about Assumed Density Filtering (ADF) and Expectation Propagation in the context of bayesian deep neural networks. I have seen in some textbooks and papers that ADF is also called moment ...
user avatar
0 votes
2 answers
145 views

How to adapt a linear time Newton-Raphson numerical method for an optimisation problem with positivity constraints?

I would appreciate some assistance in understanding how I can adapt a linear time Newton-Raphson root finding algorithm for unconstrained optimisation, to solve a problem where I introduce positivity ...
user avatar
  • 2,206
1 vote
1 answer
29 views

Is there a conditional version of $\beta$-VAE?

Pretty straightforward question. I could not find any information on the existence of a "conditional $\beta$-VAE". I'm using CVAE for a regression problem and having trouble balancing KL and ...
user avatar

1
2 3 4 5
7