Questions tagged [variational-bayes]

Variational Bayesian methods approximate intractable integrals found in Bayesian inference and machine learning. Primarily, these methods serve one of two purposes: Approximating the posterior distribution, or bounding the marginal likelihood of observed data.

Filter by
Sorted by
Tagged with
3
votes
2answers
250 views

Why computing P(x,D) is simpler than P(x|D) in exponential bayesian networks?

I am reading this tutorial on variational inference and wonder why the statement in the question title which is mentioned on page 3 is true.
4
votes
0answers
378 views

How can variational inference (for LDA) be explained in layman's terms?

I am learning probabilistic topic modeling, and I am studying latent Dirichlet allocation, specifically the inference process. I found many mathematical details, but I need a layman's explanation and/...
1
vote
0answers
84 views

Model comparison using lower bound from variational approximation

I applied variational approximation for probit regression model and got the lower bound for the log marginal likelihood. When I compare models with different covariates using lower bound, I found that ...
7
votes
2answers
656 views

choosing prior parameters for variational mixture of Gaussians

I am implementing a vanilla variational mixture of multivariate Gaussians, as per Chapter 10 of Pattern Recognition and Machine Learning (Bishop, 2007). The Bayesian approach requires to specify (...
4
votes
1answer
455 views

Why do we use the mean-field approximation for variational Bayes?

I often see the mean-field approximation for Variational Bayes. I understand the independence assumption: what I don't understand is why we make that assumption. How does it help us?
0
votes
1answer
150 views

Computing Positive definite covariance matrix in Variational Bayes GMM

This question is about a part of variational Bayes problem for GMM. (more in Bishop, Pattern recognition and Machine learning, part $10.2.1$). We are looking for $q(\mu_k,\Lambda_k)$, so we have: $$ ...
58
votes
7answers
33k views

How does the reparameterization trick for VAEs work and why is it important?

How does the reparameterization trick for variational autoencoders (VAE) work? Is there an intuitive and easy explanation without simplifying the underlying math? And why do we need the 'trick'?
6
votes
1answer
134 views

$\min D_\textrm{KL}(p(x_1,\dots,x_n) \mid\mid q_1(x_1)\cdots q_n(x_n))$ gives the marginals of $p(x_1,\dots,x_n)$?

Prove or disprove: Let $p(x_1,\dots,x_n)$ be a given probability distribution over the $n$ variables $x_1, \dots,x_n$. The univariate probability distributions $q(x_1),\dots,q(x_n)$ that minimize the ...
1
vote
0answers
156 views

The likelihood of response variables in variational Bayesian probit regression

I read the paper Explaining Variational Approximations (J.T. Ormerod & M.P. Wand) and there is a part where they explain variational probit regression with auxiliary variable since the posterior ...
3
votes
1answer
46 views

help to reproduce this derivation

I have been reading this (http://www.jting.net/pubs/2007/ting-ICRA2007.pdf) paper and attempting to derive the Variational EM update equations here. So, the model is given as follows: $$ y_i \sim N(\...
1
vote
1answer
33 views

How extract an appropriate mean from a posterior distribution

I am studying deterministic inference algorithms to learn posterior probability of Gaussian distributions and we need to find the hyperparameters for the mean and variance random variables of the ...
5
votes
1answer
82 views

how does one compute expectations for non-linear functions

I am continuing my struggles with approximate Bayesian inference methods. I have a fundamental doubt about how to compute certain expectations that arise during variational bayes, for example. So, my ...
1
vote
0answers
75 views

conjugate prior for my model parameter

This question is related to the another thread that I posted: Help with Variational Bayes on a weighted linear regression model To reiterate, I have the model as follows: $$ y_i \sim \mathcal{N}(T(...
3
votes
1answer
624 views

Bayesian inference for power laws

Let $G$ be a graph with $N$ nodes. Let $p(d_i)$ be the probability of node $i$ to have $d$ connections. If this follows a power-law: $$ p(d_i) = \frac{d_i^\alpha}{\sum_{j=1}^{N} d_j^\alpha} $$ $\...
5
votes
1answer
220 views

Help with Variational Bayes on a weighted linear regression model

I am trying to setup VB to do a weighted linear regression for vector observations. My setup is that I have $N$ numbers of $d$-dimensional vector observations. I would like to model the noise as being ...
3
votes
3answers
161 views

Applying variational inference to this model

I am basically trying to do a weighted linear regression in a bayesian way. This is to ensure that the I can take care of the heretoscedastic noise. So, my model is like: $$ y_i \sim \mathcal{N}(\...
4
votes
1answer
884 views

Variational Bayes: Understanding Mean field approximation

I am looking at the mean field approximation as used in Variational Bayes inference and I looked at this section on wikipedia with the factorised approximation as described here: https://en.wikipedia....
9
votes
1answer
218 views

Variational Bayes combined with Monte Carlo

I'm reading up on variational Bayes, and as I understand it, it comes down to the idea that you approximate $p(z\mid x)$ (where $z$ are the latent variables of your model and $x$ the observed data) ...
3
votes
3answers
2k views

Topic modeling (LDA) gives different outputs

I am using Topic Modeling Tool which is based on Mallet and using latent dirichlet allocation (LDA). When I ran the tool multiple times, with the same input (a folder of 200-500 short text files), and ...
6
votes
1answer
2k views

variational inference with KL

i am self-studying variational inference - and in Murphy's book "A probabilistic perspective on machine learning" it is discussed that minimizing the forward KL divergence (which is stated to be zero-...
4
votes
2answers
177 views

A slick expectation calculation: how was it done?

$\newcommand{\Cov}{\mathrm{Cov}}$ We have that $$ \Delta_{\lambda} KL(\lambda) = \mathbb{E}_{\theta\sim q_{\lambda}(\theta),z\sim g_{N}(z|\theta)}(\Delta_{\lambda}[\log \: q_{\lambda}(\theta)](\log \...
0
votes
1answer
326 views

normalization constant for categorical distribution as exponential family

Let r.v. $X$ has categorical distribution. We can represent its pmf as $f(x\mid\vec{p})=\Pi_{i=1}^{K}p_i^{I[x=i]}=\exp[\sum_{i=1}^{K}I[x=i]\ln p_i]$, there is no explicit normalization constant (...
1
vote
0answers
46 views

A question on notation in variational message passing

This paper introduces variational message passing. Formula (8) is based on Fig 1. Formula (a) is $\ln Q^*_j(H_j)=\langle\ln P(H_j\mid\vec{pa_j})\rangle_{\sim Q(H_j)}+\sum_{k\in ch_j}\langle\ln P(X_k\...
2
votes
0answers
50 views

Factorizable time evolution in a dynamic stochastic process

I have a stationary dynamic system which at each given time $t$ is in state $x_t \in \mathcal{X}$. The set of states $\mathcal{X}$ is assumed to be finite but too large to be enumerated by a practical ...
1
vote
0answers
34 views

computing expectations in variational updates

I have a complete log-likelihood expression as follows: $$ L = \sum_{i=1}^N \log P(y_i|x_i, w_i, \beta) + \log P(\beta) + \sum_{i=1}^N \log P(w_i) $$ Now, I need to compute the expectation of these ...
1
vote
1answer
2k views

How does the number of components in a GMM relate to the information content?

Say you fit a Gaussian Mixture Model (GMM) to your data using a Bayesian technique, which should tell you the number of components needed to fit your data. Does this also give insight into the ...
3
votes
0answers
48 views

Prior on sum of bernoulli variables

I wish to model my data as follows: $$ y\sim\mathcal{N}(X\beta,\sigma^2)\\ \beta_i\sim\mathcal{N}(5,1)^{z_i}\mathcal{N}(0,1)^{1-z_i}\\ z_i\sim logit(\gamma_i)\\ \gamma_i\sim\mathcal{N}(\gamma_{i-1},1)...
3
votes
1answer
444 views

Mean field variational inference

In Chris Bishop PRML book p.465 equation 10.6, the derivation doesn't explain why exactly the term $\int q_j ln(q_j) dz_j $ was generated, is not that term supposed to be multiplied by constant, did ...
0
votes
1answer
542 views

Reparameterization of probability distribution (spike and slab)

I try to understand a statement in this paper: http://papers.nips.cc/paper/4305-spike-and-slab-variational-inference-for-multi-task-and-multiple-kernel-learning.pdf In particular, I am talking about ...
1
vote
0answers
213 views

Find expectation or lower bound of log erf

I need to find the expectation of $\log \Phi(x)=\log \left(\int_{-\infty}^x\frac{1}{2\pi}\exp(-\frac{1}{2}s^2)ds\right)$. (I realise this isn't quite the error function, but not sure what to call it). ...
3
votes
0answers
222 views

Expectation-Maximization with dependent latent variables

Deriving the equations for a Expectation Maximization over my model, I end up with a posterior for the latent variables (E-step) that prevents me from going on. Generative model I assume my data is ...
0
votes
1answer
137 views

Backward message passing in variational Bayesian inference

I have come across in a research paper that, I do understand the logic. But the paper has't mentioned about the way of updating $\eta_{t}$. When I asked from the authors they said when we equate the ...
1
vote
1answer
263 views

Varational inference - coordinate ascent question

I see in this document on variational inference that (p.5): $L_k = \int q(z_k)E_{-k}[\log(p|z_{-k},x)]dz_k - \int q(z_k)\log q(z_k)dz_k$ It is stated that taking the derivative with respect to $q(...
6
votes
0answers
145 views

Is this problem Bayesian? And can I use variational approximation?

Suppose there are $N$ samples of observations $\mathbf X(n)$ ($n=1,\cdots,N$), which are given by probability distribution $p(\mathbf X(n)|\mathbf Z(n))$ with their conditions are given by hidden ...
6
votes
0answers
577 views

Estimation of log-likelihood via importance sampling

I am looking at a model trained with stochastic gradient variational Bayes. In this paper an importance sampler is proposed to estimate the likelihood: $$p(x) \approx {1 \over S} \sum_{s=1}^S {p(x|...
1
vote
1answer
42 views

Do I need to care about constants in Expectation Propagation

I am trying to approximate a certain factor in my graph. Following Tom Minka's tutorial what I have to do is as follows: $$ \prod_{i=1}^3 q_{w_i}(\pi_2)\approx \int p(\pi_2|w_1)q_{\pi_1}(w_1)\prod_{...
26
votes
1answer
6k views

Relation between variational Bayes and EM

I read somewhere that Variational Bayes method is a generalization of the EM algorithm. Indeed, the iterative parts of the algorithms are very similar. In order to test whether the EM algorithm is a ...
2
votes
1answer
102 views

help with this gradient computation in Expectation Propagation

I am trying to use Expectation propagation (EP) for approximating a posterior distribution in the Gaussian family. In this case, it is done by finding the Gaussian distribution with the same first and ...
8
votes
1answer
737 views

Variational inference engines

After doing some research on the topic, I have noticed a surprising deficit of inference packages and libraries that rely on message-passing or optimization methods for Python and R. To the best of ...