Questions tagged [approximate-inference]

The tag has no usage guidance.

0
votes
0answers
26 views

Are there connections between Maximum Entropy and Variational Inference?

I would like to ask what if there are any connections between Maximum Entropy and Variational Inference? I suspect that they are related somehow.
0
votes
0answers
251 views

Approximating expectation with Taylor series

I want to get the second-order Taylor approximation for an expectation. I have the following distribution, which is a Generalized Dirichlet distribution with parameters $\boldsymbol\alpha$ and $\...
0
votes
0answers
18 views

how least square estimation can be done for a distribution

As i have estimated parameters of geometric distribution by using MLE (maximum likelihood estimation) and MOM( Method of moment) but i have problem in estimating parameter of Geometric distribution ...
7
votes
0answers
157 views

Rao-Blackwellization in variational inference

The Black box VI paper introduces Rao-Blackwellization as a method to reduce the variance of the gradient estimator using score function, in section 3.1. However I don't quite get the basic idea ...
1
vote
0answers
32 views

Neyman–Pearson lemma in hypothesis testing [closed]

as i have been working for hypothesis testing , i found a NPL(Neyman–Pearson lemma) i know about all things like non-randomized and randomized test , most powerful test , critical region etc. but i am ...
0
votes
1answer
62 views

Variational Inference - deriving coordinate update equations for mixture model

I'm currently going through this paper by Blei et. al. that describes the setup and derivation of the coordinate ascent equations for a Gaussian mixture model with K components. I am having some ...
0
votes
0answers
30 views

Approximation of the upper bound on the expectation of log sum of exponentials

I am having some trouble replicating the results in Guillaume Bouchard's paper, Efficient Bounds for the Softmax Function and Applications to Approximate Inference in Hybrid Models, where he discusses ...
3
votes
1answer
72 views

variational inference derivation

According to this lecture note, Eq. 25 gives the coordinate ascent update for latent variable $z_k$ as follows $$q^*(z_k)\propto\exp(E_{-k}[\log{p(z_k,Z_{-k},x)}])$$ and I understand the derivation ...
0
votes
0answers
16 views

Why is the integral of the partial factorized distribution equal to one on the derivation of the variational inference lower bound?

Considering Variational Inference - factorized distributions and evidence lower bound, a proposed answer solution indicates that the integral of the partial factors equals to one $$ \int \Pi_{j \neq i}...
2
votes
1answer
74 views

Questions about approximate inference and calculating the posterior predictive

As I understand, computing the exact posterior of parameters $p(\theta|x)$ is nearly always impossible since we need to compute the evidence $\sum_\theta p(x|\theta)p(\theta)$ with every possible ...
2
votes
1answer
136 views

Variational Inference: Ising Model

I am self learning Variational Inference. Currently I am reading the chapter 21 book from Murphy 1 and trying to understand the Ising model (21.3.2). The Ising model here is used as denoising ...
1
vote
0answers
58 views

Efficient approximate marginal inference in variational auto-encoder

In Auto-Encoding Variational Bayes, authors mentioned that they proposed a solution to "Efficient approximate marginal inference of the variable $x$". I read through the paper and appendix, now ...
0
votes
0answers
26 views

Analytical solution of $D_{KL}(q_\phi (z) || p_\theta (z))$

In appendix of "Auto-Encoding Variational Bayes" by Kingma & Welling, they provide a solution of $D_{KL}(q_\phi (z) || p_\theta (z))$ for Gaussian case. I feel confused about the notation for the ...
1
vote
0answers
70 views

Jensen's inequality in Collaborative Topic Regression

I am reading the article Collaborative Topic Modeling for Recommending Scientific Articles and could notice the application of Jensen's inequality in order to define a bound from which optimization is ...
4
votes
1answer
108 views

How Can I teach someone “sampling from a given distribution” is hard?

For many people I know, they do not think sampling from a given distribution is a hard problem in general. For example, many software provide functions do to sample from normal distribution or uniform ...
2
votes
1answer
156 views

Variational Inference of Univariate Gaussian mixtures

I am reading this paper. In the paper, they use an example of mixture of unit-variance univariate Gaussians with the following parameterization: \begin{align} \mu_k & \sim \mathcal{N}(0, \sigma^2)...
2
votes
1answer
208 views

Computing the gradient of the expectation of a function w.r.t the parameters of the distribution

I am reading this paper on variational auto-encoders by Kingma & Welling and in section 2.2 authors write the following equality for the gradient of the expectation of a function with respect to ...
1
vote
1answer
667 views

Difference between stochastic variational inference and variational inference?

Very simple, as the question header says: what is the difference between SVI and VI? I cannot seem to find a clear-cut definition online.
2
votes
1answer
338 views

Why does detailed balance not provide a stopping criterion in MCMC?

Like I undestand MCMC sampling, the fulfillment of the detailed balance equation guarantees that our MC has reached its stationary distribution (given we ensure ergodicity). Detailed Balance is: $\...
0
votes
1answer
120 views

Normalizing Flows, Real NVPs and Inverse Autoregressive Flows - Used for Probabilty Density Approximation or for Sampling?

Suppose we have a parametric family $g(x;\theta)$, where $\theta$ are the parameters. As far as I can tell, there are two ways we can use this family to model a probability distribution: Probability ...
3
votes
0answers
269 views

Convergence of approximate Gibbs sampling

We have a bivariate random variable $(X,Y)$ for which sampling is challenging. If we were to know how to sample from the conditionals $(X|Y)$ and $(Y|X)$, we could get samples from the joint using ...
1
vote
2answers
777 views

get probabilities from kernel density estimation pdf

I have data points located at $\mathbf{x}_i$ and I would like to a find quick and dirty way to calculate their probability of occurring (not the pdf) using kernel density estimation. Formally speaking,...
1
vote
1answer
76 views

Bethe approximation for factor graphs

I am confused at computing Bethe approximation for factor graphs in here. It generalizes Bethe approxmiation in a pairwise case. However, I am wondering why (75) goes to (78) with (76): We can verify ...
0
votes
1answer
113 views

How to compute the Gibbs free energy in Bethe approximation for MRF

Hi, I am learning loopy belief propagation for MRF. The general roadmap is to define a Bethe approximation, which is exact for a tree but wrong for general graphs. I'm currently stuck at the point to ...
3
votes
0answers
60 views

Simple approximation of joint posterior

Consider the (hierarchical) Bayesian inference problem with two unknowns $(x,\theta)$ and data $y$. I'm using a very simple ("independence"?) approximation $$ p(x,\theta|y) \approx p(x|\theta_\star,y) ...
5
votes
2answers
508 views

Estimating the gradient of log density given samples

I am interested in estimating the gradient of the log probability distribution $\nabla\log p(x)$ when $p(x)$ is not analytically available but is only accessed via samples $x_i \sim p(x)$. There ...
31
votes
1answer
5k views

Variational inference versus MCMC: when to choose one over the other?

I think I get the general idea of both VI and MCMC including the various flavors of MCMC like Gibbs sampling, Metropolis Hastings etc. This paper provides a wonderful exposition of both methods. I ...
5
votes
1answer
746 views

Explanation of the 'free bits' technique for variational autoencoders

I have been reading through a couple of papers on the variational autoencoder model: 'Variational Lossy Autoencoder' and 'Improving Variational Inference With Inverse Autoregressive Flow'. There is ...
5
votes
2answers
977 views

Maximum likelihood estimator that is not a function of a sufficient statistic

I always read that every maximum likelihood estimator has to be a function of any sufficient statistic. The idea is that, if we are dealing with a random variable $X$ with mass or density function $f(...
4
votes
1answer
512 views

What are the advantages of normalizing flow over VAEs with deep latent gaussian models for inference?

I am reading the normalizing flow paper and am a bit confused. It seems that being able to model complex (correlated?) posterior is one of the advantages of the proposed approach (Section 2.3, last ...
1
vote
0answers
36 views

Deriving the particle filter with driving-force/inputs/control-signal

Whenever the particle filter is derived (I used a different condition for $u_t$ as a solution to the nonlinear filtering problem; $x_{t+1} \mid x_t \sim f_{\theta}(x_{t+1} \mid x_t,u_t) \\ y_{t} \...
12
votes
3answers
2k views

what does one mean by numerical integration is too expensive?

I am reading about Bayesian inference and I came across the phrase "numerical integration of the marginal likelihood is too expensive" I do not have a background in mathematics and I was wondering ...
3
votes
1answer
130 views

Examples where the evaluation of the posterior distribution $p(Z|X)$ of the latent variables $Z$ is a central task?

In PRML Chapter 10 Approximate Inference, the fist sentence of the chapter says A central task in the application of probabilistic models is the evaluation of the posterior distribution $p(Z|X)$ of ...
1
vote
0answers
984 views

Bayesian Networks - CPD representation and inference for non-Gaussian continuous variables

I'm trying to implement an approximate inference algorithm based on junction tree algorithm for a Bayesian Network that has continuous variables which happen to have non-linear relationships, and in ...
1
vote
0answers
100 views

Expectation Propagation - Computing mean and variance of error function

I'm still trying to wrap my head around computing the moments for the expectation propagation algorithm and whether I can use it for the following example: say i have a product of distributions which ...
6
votes
1answer
551 views

How is ABC more computationally efficient than exact Bayesian Computation for parameter estimation in dynamical systems (ODE) models?

Approximate Bayesian Computation has been suggested as an approach to parameter estimation for computationally intensive simulations, most commonly in population genetics, but also in dynamical ...
1
vote
1answer
77 views

Bayesian inference of marginal likelihood using ABC

I have the following situation: suppose data $D = \{x_i\}$ iid are generated through some process with density function $f(x_i | \alpha, \beta)$ (which I think will be negative binomial) and we'd like ...