Questions tagged [variational-bayes]

Variational Bayesian methods approximate intractable integrals found in Bayesian inference and machine learning. Primarily, these methods serve one of two purposes: Approximating the posterior distribution, or bounding the marginal likelihood of observed data.

Filter by
Sorted by
Tagged with
0
votes
0answers
7 views

Variational inference - posterior predictive distribution

So suppose we have a neural network that aims to map values from one distribution to another. That is to say the inputs do not belong to the same distribution as the targets. It then follows that, ...
1
vote
1answer
57 views

Variational Bayes for a univariate Gaussian

I'm following an example from Murphy's book (Sec 21.5.1) on how to apply Variational Bayes to infer the posterior over the parameters for a 1D Gaussian $p(\mu,\lambda|\mathcal{D})$. The example uses a ...
0
votes
0answers
19 views

Variational inference: Rewriting ELBO

I was following some tutorials on variational inference and ELBO loss functions and I think I understand it quite well, but I'm struggling with math. If I understand it correctly we went from this ...
0
votes
0answers
29 views

Is multicollinearity a problem when fitting a Bayesian regression model using ADVI?

If I’m fitting a bayesian regression model using ADVI, is it important to ensure all the covariates are uncorrelated with each other? I have a vague understanding that ADVI doesn’t play well with ...
0
votes
0answers
8 views

Evaluating evidence lower bound for Poisson factorisation model

tl;dr This question is about evaluating the evidence lower bound (ELBO) for variational inference in the context of count data. Background I am trying to reproduce hierarchical Poisson factorisation ...
1
vote
1answer
15 views

forward and reverse KL divergence for variational inference

I have a question regarding the forward or reverse KL divergence used in variational inference. In accordance with the following lecture notes, reverse KL can cause q under-estimate the support of p ...
0
votes
1answer
13 views

Stochatic Variational Bayes with distribution on parameters?

In Autoencoding Variational Bayes, the authors show, in the Appendix F, that SGVB can be performed with a model where we have a distribution over the generative parameters \theta (Full VB). They ...
0
votes
1answer
34 views

Marginal likelihood: Why is it difficult to compute in this case?

I have been reading up a bit on generative models particularly trying to understand the math behind VAE. While looking at a talk online, the speaker mentions the following definition of marginal ...
0
votes
0answers
45 views

How to optimize a function with respect to a distribution, in the context of variational inference

Context: I am learning about variational inference. The reference I am following is linked at the end of this post. Goal: I want to learn how a marginal variational distribution $q_k$ is optimal in ...
0
votes
0answers
14 views

Reference Request: Variational Expectation-Maximization algorithm for Latent Dirichlet Allocation with an added time component

This link has a pretty good runthrough on the variational inference (via variational E-M) for LDA with calculations expanded and explained. I am now considering a modified LDA which adds a time ...
2
votes
2answers
49 views

Inference: How is the Laplace approximation actually useful to us compared with MLE and MAP?

I was reading a few different sources (including the "Machine Learning and Pattern Recognition" book by Bishop) about the Laplace integral approximation method for inference. However, I am ...
0
votes
0answers
20 views

Algorithm of variational autoencoders

Recently I was reading about variational autoencoders and I'm not sure if I correctly understand the algorithm of it. I also met very good lecture on you tube about autoencoders and there I found the ...
2
votes
1answer
90 views

Semi-supervised classification objective from Kingma et al

In this 2014 paper, Kingma et al. develop different methods to do semi-supervised learning with VAEs. In one of their proposed solutions ("M2"), they approach this problem by incorporating ...
2
votes
0answers
25 views

How to derive local variational close form for logistic regression, analytically

In Prof Bishop's book, Pattern Recognition and Machine Learning, Chapter 10, on local variational inference for logistic regression, page 501, equation 10.161 has been derived, from differentiating EM ...
0
votes
0answers
7 views

Understanding variational inference for Bayesian GMM

I am reading Variational Inference: A Review for Statisticians by Blei, Kucukelbir and McAuliffe. I am having a hard time following some of the steps in Section 3.2 More precisely, they state that $\...
0
votes
1answer
49 views

Regression in Baysian settings

Assume we have the posterior distribution of this linear regression model $y = w^Tx$, $P(w | D,\theta)$, where $D = \{(x_i,y_i)\}_{i \in \{1,\dots,n\}}, n $ is the number of data instances, $\theta$ ...
0
votes
0answers
31 views

Can a bayesian neural network with independent normal variational distributions over weights and biases produce a multi-modal posterior?

For a project I am involved in, I am doing surrogate modelling. This means that I simulate data $\mathcal{D}=\{X,Y\}$ that is used to train some probabilistic non-linear regression model. The model is ...
1
vote
0answers
19 views

The Role Variational Free Energy or Expected Lower Bound (ELBO) Plays as a Loss Function

In reference to variational free energy or expected lower bound, I found this sentence, "As one can easily see, the cost function tries to balance the complexity of the data P(D | w) and the ...
0
votes
2answers
50 views

In bayesian approach, what is the difference between full posterior and MAP [duplicate]

Consider a classic machine learning problem, which we want to solve using NN. And suppose that we want to use bayesian learning for that. In the bayesian approach the posterior is described as follows:...
1
vote
1answer
74 views

Clarification of Equation for Variational Inference in Pattern Recognition and Machine Learning

I am looking at the derivation of variational inference and specifically the approach taken by Bishop in his book on page 465 as illustrated in the Figure below. The key step is the statement below ...
0
votes
1answer
29 views

Is outer product of marginal distribution the "best" mean-field approximation for a joint distribution?

I am certain this has been asked somewhere else, if that's the case, link me and close the thread. I am studying variational inference and mean-field approximation. All the explanations I come across ...
2
votes
1answer
41 views

In VAE, why are we approximating p(z|x) using q(z) and not q(z|x) [duplicate]

I am watching this lecture on VAE: https://www.youtube.com/watch?v=uaaqyVS9-rM&t=1507s and at 26:00, it is stated that the goal is the minimize the KL div. between the distribution we are trying ...
1
vote
1answer
29 views

Approximating the posterior and learning the distribution over the weights after training

I am familiar with the methods in variational inference in which after training we have access to the distribution over the network's weights. This is necessary for estimating epistemic uncertainty. ...
3
votes
1answer
135 views

How do we get to the MSE in the loss function for a variational autoencoder?

Context: https://arxiv.org/pdf/1312.6114.pdf So if I start with this equation: $$ \mathcal{L}\left(\boldsymbol{\theta}, \boldsymbol{\phi} ; \mathbf{x}^{(i)}\right) \simeq \frac{1}{2} \sum_{j=1}^{J}\...
0
votes
0answers
51 views

Convergence of CAVI(Coordinate Ascent Variational Inference)

I was reading several resources on variational inference, and most of them stated that the CAVI algorithm converges to local maximum, and Bishop's textbook stated that the convergence is guaranteed as ...
1
vote
0answers
56 views

I am confused about the sum indices of these posterior distribution formulas

I am reading this notes and trying to understand how the posterior distribution formulas of the variational variables involved have been calculated. I am confused about the indices of the summation. ...
1
vote
0answers
23 views

Tied Bayesian Mixture of Gaussians

I am bit confused when it comes to modelling a Bayesian Gaussian mixture model that assumes a shared covariance/precision matrix for all Gaussian components. I followed the derivation in Bishop and ...
1
vote
1answer
118 views

Why do we start from $KLD(q(z) || p(z|x))$ in ELBO derivation?

Of the several ways to derive the evidence lower bound (such as using Jensen’s inequality), a version often used is the derivation from $KLD(q(z) || p(z|x))$. The following image illustrates the ...
1
vote
0answers
62 views

Is there a reason to use variational inference for point estimates?

I have seen Bayesian hierarchical models, particularly in computational biology, that use variational inference, but do not use the uncertainty provided by a variational solution. For example, MOFA is ...
0
votes
0answers
18 views

Moments of Linearly Transformed Laplace Distribution and Assumed Density Filtering

I have been occupied with a question that I assume is not as difficult as I find it to be. The question I want to solve boils down to finding the moments of a linearly transformed Laplace distribution....
0
votes
0answers
29 views

Is Beta-VAE regularization equivalent to using a Gaussian prior with variance smaller than 1?

The Beta-VAE model offers the regularization over the -$\beta$ KL(q(z|x)||p(z)) term of the ELBO term in VAE formulation. Scaling this term up forces some latents to be distributed as unit Gaussians (...
0
votes
0answers
21 views

the variational family used to approximate the weight posterior of a BNN

Why the variational family used to approximate the weight posterior of a BNN is often chosen to be Gaussian?
0
votes
0answers
23 views

Weight Uncertainty in Bayesian Neural Networks

In Variational inference for deep neural networks, weights can be represented with a distribution. This distribution has parameters. In the case of Gaussian, we have mean and variance as the ...
0
votes
3answers
197 views

Why Assumed Density Filtering is also called Moment Matching?

I am learning about Assumed Density Filtering (ADF) and Expectation Propagation in the context of bayesian deep neural networks. I have seen in some textbooks and papers that ADF is also called moment ...
0
votes
0answers
17 views

Amortized inference in convolutional variational autoencoders

VAEs are an efficient way of performing variational inference at scale. I read that VAEs employ the strategy of amortized variational inference. They approximate the intractable posteriors p(zjx) by ...
0
votes
2answers
124 views

How to adapt a linear time Newton-Raphson numerical method for an optimisation problem with positivity constraints?

I would appreciate some assistance in understanding how I can adapt a linear time Newton-Raphson root finding algorithm for unconstrained optimisation, to solve a problem where I introduce positivity ...
1
vote
1answer
26 views

Is there a conditional version of $\beta$-VAE?

Pretty straightforward question. I could not find any information on the existence of a "conditional $\beta$-VAE". I'm using CVAE for a regression problem and having trouble balancing KL and ...
0
votes
0answers
29 views

Why is this integral equal to $1$? (VBIL)

Let $p(y \mid \theta)$ be a likelihood and $\hat{p}_N(y \mid \theta)$ be an unbiased estimator of it. In VBIL they define $z = \log \hat{p}_N(y \mid \theta) - \log p(y\mid \theta)$ and call its ...
4
votes
2answers
82 views

Why does it appear impossible to fit Gaussians to arbitrary probability density functions $p$?

I want to fit a Gaussian $q$ to a pdf $p$ by minimizing the energy $E = -\int q(x) \log p(x) dx$. This should result in a "delta function" Gaussian with $\sigma \rightarrow 0$ and $\mu \...
0
votes
1answer
34 views

Why is the posterior chosen to be normal in variational autoencoder?

Is there any reason for choosing the posterior 𝑞(𝑧|𝑥) as normal distribution in variational autoencoder? or is it just for convenience?
1
vote
1answer
68 views

Does variational inference solve the label switching problem in Bayesian mixture estimation?

This paper (p. 1751) claims that variational methods do not suffer the label switching problem inherent to Bayesian estimation of mixture models. However, I struggle to find additional references and ...
1
vote
0answers
54 views

Online Stochastic Variational Inference for Dirichlet Process Mixture Models

There's a 2013 NeurIPS paper I'm trying to understand, Online Learning of Nonparametric Mixture Models via Sequential Variational Approximation. I have a few questions: Equation 2, which defines a ...
0
votes
0answers
20 views

Practical ways to measure uncertainty in deep neural network for classification

I have a deep neural network for classification task. The training set contains rich feature vectors X and binary labels Y. The goal of the network is given X, predict the probability for each class. ...
6
votes
1answer
208 views

Mathematical notation for suppressing differentiation

Basic question Is the some existing mathematical notation to mean "treat this term as a constant when differentiating"? This would be the equivalent of ...
2
votes
1answer
161 views

Measuring uncertainty with bayesian neural network

One of the ways to measure epistemic uncertainty, is using bayesian inference in neural networks. The idea is to learn the posterior over the weights $P(\phi|X)$ which describe the probability ...
0
votes
0answers
28 views

Question regarding ELBO value of a VAE

I'm currently training a VAE (Which I'm a beginner) and have a question about the ELBO. Can you approximate the empirical probability of a specific data point 'x' in the dataset using the evidence ...
0
votes
1answer
94 views

Why is joint probability assumed to be trivially calculable in variational inference?

The ELBO in variational inference can be written as: $\sum q(z)\log \frac{p(x,z)}{q(x)}$. In videos describing variational inference, I often see how lecturers decribe "joint distributions ($p(x,...
0
votes
0answers
29 views

Robustness of ELBO

The ELBO $\mathcal{L}(\phi)$ is used to quantify how good an approximate posterior $q_\phi(z|x)$ is for a dataset $x$ and an (unknown) true posterior $p_\theta(z|x)$. However this is all under the ...
0
votes
2answers
44 views

Conditional vs. nonconditional variational family

In variational bayes, distributions in the variational family $\mathcal{Q}$ are denoted $q_\phi$ and are used to approximate the posterior $p_\theta(z|x)$. However, I've seen both notations $q_\phi(z|...
0
votes
0answers
16 views

Deep Bayesian networks learning techniques

I am trying to compare different learning techniques to train deep Bayesian neural networks. do you have any suggestions or papers that do compare different learning techniques such as mean-field ...

1
2 3 4 5
7