Questions tagged [approximate-inference]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
3
votes
0answers
63 views
+300

How to Test Linear Hypotheses about Parameters in Simulation-Based Indirect Inference

Setup: I have a model that produces a vector of aggregate outcomes, $\theta$, based on parameters, $\beta$. The relationship $\theta=\Theta(\beta)$ is stochastic and analytically intractable, but I ...
0
votes
1answer
47 views

What is the difference between approximate bayesian computation vs approximate bayesian inference?

What are the main differences between approximate bayesian computation vs approximate bayesian inference? Are they essentially the same? Do they refer to the same of different family of models? My ...
0
votes
1answer
15 views

How to combine sampled data from the same population?

Let's say I have a friend and we both asked one group of people a different question. For example, I ask the group how old they are, and my friend asks them how much they weigh. If I meet up with my ...
1
vote
0answers
69 views

Computing gradient of KL-divergence

I'm trying to compute the gradient of the following quantity with respect to $\mathbf{w} \in \mathbb{R}^N$ based on the KL divergence between two normal distribution $\mathrm{KL}(N(\boldsymbol{\mu}_w,\...
0
votes
0answers
38 views

Is “variational approximation” synonymous with “variational inference”?

The title says it all. I am currently reading up on deep generative models, and frequently encounter the term "variational inference" as well as the term "variational approximation" to refer to what ...
0
votes
1answer
74 views

Variational inference with discrete variational parameters

Typically Variational Inference relies on taking gradient steps on KL divergence between the variational and true posterior, or on the ELBO. This does not seem valid when variational parameters are ...
1
vote
1answer
95 views

Variational Inference with intractable score function

Is it possible to do ELBO maximization using stochastic gradient estimates (i.e. iteratively apply variational updates using (3) in http://proceedings.mlr.press/v33/ranganath14.pdf), when it's cheap ...
2
votes
1answer
88 views

Variational inference with deterministic dependencies between variables

Suppose I have a probabilistic graphical model shown in the picture, in which all variables are binary, $c_1$ and $c_2$ are observed, and I want to use mean-field variational inference to estimate ...
0
votes
0answers
35 views

Are there connections between Maximum Entropy and Variational Inference?

I would like to ask what if there are any connections between Maximum Entropy and Variational Inference? I suspect that they are related somehow.
0
votes
0answers
270 views

Approximating expectation with Taylor series

I want to get the second-order Taylor approximation for an expectation. I have the following distribution, which is a Generalized Dirichlet distribution with parameters $\boldsymbol\alpha$ and $\...
0
votes
0answers
27 views

how least square estimation can be done for a distribution

As i have estimated parameters of geometric distribution by using MLE (maximum likelihood estimation) and MOM( Method of moment) but i have problem in estimating parameter of Geometric distribution ...
7
votes
0answers
196 views

Rao-Blackwellization in variational inference

The Black box VI paper introduces Rao-Blackwellization as a method to reduce the variance of the gradient estimator using score function, in section 3.1. However I don't quite get the basic idea ...
0
votes
1answer
88 views

Variational Inference - deriving coordinate update equations for mixture model

I'm currently going through this paper by Blei et. al. that describes the setup and derivation of the coordinate ascent equations for a Gaussian mixture model with K components. I am having some ...
0
votes
0answers
37 views

Approximation of the upper bound on the expectation of log sum of exponentials

I am having some trouble replicating the results in Guillaume Bouchard's paper, Efficient Bounds for the Softmax Function and Applications to Approximate Inference in Hybrid Models, where he discusses ...
3
votes
1answer
80 views

variational inference derivation

According to this lecture note, Eq. 25 gives the coordinate ascent update for latent variable $z_k$ as follows $$q^*(z_k)\propto\exp(E_{-k}[\log{p(z_k,Z_{-k},x)}])$$ and I understand the derivation ...
2
votes
1answer
109 views

Questions about approximate inference and calculating the posterior predictive

As I understand, computing the exact posterior of parameters $p(\theta|x)$ is nearly always impossible since we need to compute the evidence $\sum_\theta p(x|\theta)p(\theta)$ with every possible ...
2
votes
1answer
184 views

Variational Inference: Ising Model

I am self learning Variational Inference. Currently I am reading the chapter 21 book from Murphy 1 and trying to understand the Ising model (21.3.2). The Ising model here is used as denoising ...
1
vote
0answers
70 views

Efficient approximate marginal inference in variational auto-encoder

In Auto-Encoding Variational Bayes, authors mentioned that they proposed a solution to "Efficient approximate marginal inference of the variable $x$". I read through the paper and appendix, now ...
1
vote
0answers
82 views

Jensen's inequality in Collaborative Topic Regression

I am reading the article Collaborative Topic Modeling for Recommending Scientific Articles and could notice the application of Jensen's inequality in order to define a bound from which optimization is ...
4
votes
1answer
109 views

How Can I teach someone “sampling from a given distribution” is hard?

For many people I know, they do not think sampling from a given distribution is a hard problem in general. For example, many software provide functions do to sample from normal distribution or uniform ...
2
votes
1answer
195 views

Variational Inference of Univariate Gaussian mixtures

I am reading this paper. In the paper, they use an example of mixture of unit-variance univariate Gaussians with the following parameterization: \begin{align} \mu_k & \sim \mathcal{N}(0, \sigma^2)...
3
votes
1answer
546 views

Gradient of the expectation of a function w.r.t. distribution parameters

In section 2.2 of Kingma & Welling's paper on variational auto-encoders authors write the following equality for the gradient of the expectation of a function with respect to the parameters of the ...
1
vote
1answer
964 views

Difference between stochastic variational inference and variational inference?

Very simple, as the question header says: what is the difference between SVI and VI? I cannot seem to find a clear-cut definition online.
2
votes
1answer
374 views

Why does detailed balance not provide a stopping criterion in MCMC?

Like I undestand MCMC sampling, the fulfillment of the detailed balance equation guarantees that our MC has reached its stationary distribution (given we ensure ergodicity). Detailed Balance is: $\...
0
votes
1answer
147 views

Normalizing Flows, Real NVPs and Inverse Autoregressive Flows - Used for Probabilty Density Approximation or for Sampling?

Suppose we have a parametric family $g(x;\theta)$, where $\theta$ are the parameters. As far as I can tell, there are two ways we can use this family to model a probability distribution: Probability ...
3
votes
0answers
311 views

Convergence of approximate Gibbs sampling

We have a bivariate random variable $(X,Y)$ for which sampling is challenging. If we were to know how to sample from the conditionals $(X|Y)$ and $(Y|X)$, we could get samples from the joint using ...
1
vote
2answers
1k views

get probabilities from kernel density estimation pdf

I have data points located at $\mathbf{x}_i$ and I would like to a find quick and dirty way to calculate their probability of occurring (not the pdf) using kernel density estimation. Formally speaking,...
1
vote
1answer
126 views

Bethe approximation for factor graphs

I am confused at computing Bethe approximation for factor graphs in here. It generalizes Bethe approxmiation in a pairwise case. However, I am wondering why (75) goes to (78) with (76): We can verify ...
0
votes
1answer
131 views

How to compute the Gibbs free energy in Bethe approximation for MRF

Hi, I am learning loopy belief propagation for MRF. The general roadmap is to define a Bethe approximation, which is exact for a tree but wrong for general graphs. I'm currently stuck at the point to ...
3
votes
0answers
62 views

Simple approximation of joint posterior

Consider the (hierarchical) Bayesian inference problem with two unknowns $(x,\theta)$ and data $y$. I'm using a very simple ("independence"?) approximation $$ p(x,\theta|y) \approx p(x|\theta_\star,y) ...
6
votes
2answers
656 views

Estimating the gradient of log density given samples

I am interested in estimating the gradient of the log probability distribution $\nabla\log p(x)$ when $p(x)$ is not analytically available but is only accessed via samples $x_i \sim p(x)$. There ...
37
votes
1answer
6k views

Variational inference versus MCMC: when to choose one over the other?

I think I get the general idea of both VI and MCMC including the various flavors of MCMC like Gibbs sampling, Metropolis Hastings etc. This paper provides a wonderful exposition of both methods. I ...
5
votes
1answer
883 views

Explanation of the 'free bits' technique for variational autoencoders

I have been reading through a couple of papers on the variational autoencoder model: 'Variational Lossy Autoencoder' and 'Improving Variational Inference With Inverse Autoregressive Flow'. There is ...
5
votes
3answers
2k views

Maximum likelihood estimator that is not a function of a sufficient statistic

I always read that every maximum likelihood estimator has to be a function of any sufficient statistic. The idea is that, if we are dealing with a random variable $X$ with mass or density function $f(...
5
votes
1answer
710 views

What are the advantages of normalizing flow over VAEs with deep latent gaussian models for inference?

I am reading the normalizing flow paper and am a bit confused. It seems that being able to model complex (correlated?) posterior is one of the advantages of the proposed approach (Section 2.3, last ...
1
vote
0answers
37 views

Deriving the particle filter with driving-force/inputs/control-signal

Whenever the particle filter is derived (I used a different condition for $u_t$ as a solution to the nonlinear filtering problem; $x_{t+1} \mid x_t \sim f_{\theta}(x_{t+1} \mid x_t,u_t) \\ y_{t} \...
12
votes
3answers
2k views

what does one mean by numerical integration is too expensive?

I am reading about Bayesian inference and I came across the phrase "numerical integration of the marginal likelihood is too expensive" I do not have a background in mathematics and I was wondering ...
3
votes
1answer
153 views

Examples where the evaluation of the posterior distribution $p(Z|X)$ of the latent variables $Z$ is a central task?

In PRML Chapter 10 Approximate Inference, the fist sentence of the chapter says A central task in the application of probabilistic models is the evaluation of the posterior distribution $p(Z|X)$ of ...
1
vote
0answers
1k views

Bayesian Networks - CPD representation and inference for non-Gaussian continuous variables

I'm trying to implement an approximate inference algorithm based on junction tree algorithm for a Bayesian Network that has continuous variables which happen to have non-linear relationships, and in ...
1
vote
0answers
109 views

Expectation Propagation - Computing mean and variance of error function

I'm still trying to wrap my head around computing the moments for the expectation propagation algorithm and whether I can use it for the following example: say i have a product of distributions which ...
7
votes
1answer
587 views

How is ABC more computationally efficient than exact Bayesian Computation for parameter estimation in dynamical systems (ODE) models?

Approximate Bayesian Computation has been suggested as an approach to parameter estimation for computationally intensive simulations, most commonly in population genetics, but also in dynamical ...
1
vote
1answer
80 views

Bayesian inference of marginal likelihood using ABC

I have the following situation: suppose data $D = \{x_i\}$ iid are generated through some process with density function $f(x_i | \alpha, \beta)$ (which I think will be negative binomial) and we'd like ...