Skip to main content

Questions tagged [approximate-inference]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
0 votes
0 answers
85 views

Why don’t diffusion models suffer posterior collapse?

In VAEs, posterior collapse occurs when the approximated posterior $q_\theta(z|x)$ becomes the standard Gaussian prior $p(z)$ after training (Lucas et al. 2019). The forward process of diffusion ...
entropy07's user avatar
1 vote
0 answers
23 views

How do I estimate probability of someone having condition X [closed]

There is a survey, which asks following question: do you know personally someone, who has condition X? The result of the survey is percentage of people, who know someone I want to use this survey to ...
Arsen Zahray's user avatar
1 vote
1 answer
90 views

Does approximating the likelihood function violate the likelihood principle in Bayesian Inference?

Suppose we have a prior $p(\theta)$ and a likelihood function $L(\theta|x)$, and that the likelihood $L(\theta|x)$ is intractable somehow (difficult or impossible to compute) and we instead replace it ...
user356974's user avatar
1 vote
1 answer
30 views

A soft question about probabilistic inference

Suppose A and B are two propositions and $ A \implies B$ is true.WE know all it means is that B is true whenever A is true. Now consider a situation where we only that A is true with probability ,say,...
AgnostMystic's user avatar
1 vote
1 answer
103 views

Generative model that satisfies certain algebraic constraints

Disclaimer: I need guidance and help with where to start looking for solution of the problem I have described below. My background is in optimization and I am new to statistical methods, so there is a ...
s.yadegari's user avatar
0 votes
0 answers
27 views

How to approximate the expression to $\sum x_i$

How to approximate the expression on the left hand side to $\sum_{i=1}^Nx_i$ as $n\to \infty$ $$ \frac{\sum\limits_{i=1}^{N}x_i^2}{n-2\frac{\sum\limits_{i=1}^{N}x_i}{N}} \left(\sqrt{1+\frac{Nn\left(...
Sara's user avatar
  • 11
0 votes
1 answer
41 views

Regression where the matrix is drawn from a known distribution

I need to estimate the vector x, where Ax = b, given b is observed data and the columns in the matrix A are drawn from a known distribution. a and b are both formed of positive integers (or zero). For ...
Graeme's user avatar
  • 1
0 votes
2 answers
106 views

Conditional vs. nonconditional variational family

In variational bayes, distributions in the variational family $\mathcal{Q}$ are denoted $q_\phi$ and are used to approximate the posterior $p_\theta(z|x)$. However, I've seen both notations $q_\phi(z|...
900edges's user avatar
  • 399
3 votes
1 answer
1k views

Understanding the set of latent variables $Z$ in variational inference

I have been trying to understand variational inference (in a Bayesian context) where we’re trying to approximate $p(Z|X)$ where $Z$ is the set of latent variables and $X$ is the set of observable ...
Nick's user avatar
  • 33
1 vote
0 answers
56 views

Non-submodularity of sparse approximations to Gaussian posterior

Imagine that we are doing approximate inference on a Gaussian mean, minimizing a variational objective that corresponds to the reverse KL divergence between a sparse approximate posterior computed on ...
Dion's user avatar
  • 954
0 votes
1 answer
28 views

Average sales of last three years using interpolation/extrapolation

I have average sales value for the last three years of a company, e.g. year avg. sales 2017 100 2018 150 2019 200 Is it possible to somehow come up with an estimate of total average sales value ...
sameerz's user avatar
  • 31
1 vote
1 answer
838 views

Are VAE useful for Maximum-Likelihood estimation?

In the "Auto-encoding Variation Bayes" Paper they state under "2.1 Problem Scenarios" that the VAE is a solution to: "1. Efficient approximate ML or MAP estimation for the ...
Lochend's user avatar
  • 51
1 vote
0 answers
66 views

Multinomial distribution

I want to show this theorem. here you get the details of it. http://yaroslavvb.com/upload/wasserman-multinomial.pdf?fbclid=IwAR1V6yB5-100XSzk6vt5cgNzV0PHdT9qJ8mwxEGplV1q-QiwgP3-Z7i0jvA where $p$ is ...
pythonhater's user avatar
1 vote
1 answer
186 views

Introduction to approximate message passing

I'm interested in learning approximate message passing from the paper "Message Passing Algorithms for Compressed Sensing: I. Motivation and Construction". My background is in computer ...
William's user avatar
  • 117
1 vote
0 answers
30 views

Are there convenient methods/tricks to make calculations with non-independent terms? (two examples here in particular)

In a recent question it came out that I needed to calculate the sample distribution of $\dfrac{Cov(XY,X)}{Var(X)}$ where the distributions of $X$ and $Y$ are known. By this, I mean the law of $$\dfrac{...
Arnaud Mortier's user avatar
1 vote
1 answer
204 views

What is a Dirac distribution on a hyperplane?

I'm trying to understand message passing for compressed sensing. I came acrross this distribution: As the title suggests, how does this distribution look like? I know the first products term in the ...
William's user avatar
  • 117
2 votes
0 answers
68 views

What is the link between the queries Bayesian Networks can answer, and inference algorithms?

I have seen two concepts linked to Bayesian Networks: Bayesian Networks can answer different types of queries. These types include proof of evidence, most probable explanation, computing maximum a ...
constance destais's user avatar
1 vote
0 answers
37 views

Can we ignore the generation side of the method described in density estimation using Real NVP?

First appologies if my question is stupid. I am studying the paper "Density estimation using real NVP" by Dinh, Sohl-Dickstein and Bengio. link The paper presented a nice idea that the generation ...
honglei's user avatar
  • 11
0 votes
1 answer
154 views

Difference / Relationship of Generative Models / Variational Bayesian Inference

I feel a bit confused trying to merge and unify understandings of generative models and variational bayesian inference methods. Initially, I believed them to be the same thing, namely learning full ...
user avatar
2 votes
1 answer
141 views

Derivation of the Objective Function for Expectation Propagation

I was reading Expectation Propagation As A Way Of Life and the original paper by Minka Expectation Propagation for Approximate Bayesian Inference and they both say that a fixed point of the EP ...
Euler_Salter's user avatar
  • 2,236
0 votes
0 answers
38 views

Exact inference in an approximate model as opposed to approximate inference in an exact model?

I remember hearing a while ago that it was more rigorous to perform approximate inference in an exact model as opposed to exact inference in an approximate model. I can’t now remember the reasoning ...
Tom's user avatar
  • 21
1 vote
0 answers
11 views

Approximate Bayesian computation for comparing parameters affect on a response variable [closed]

I'm not sure I understand this method perfectly so please correct me if I'm wrong. From my understanding Approximate Bayesian Computation allows you to perform likelihood free inference by re-...
Jin's user avatar
  • 586
3 votes
1 answer
134 views

How to Test Linear Hypotheses about Parameters in Simulation-Based Indirect Inference

Setup: I have a model that produces a vector of aggregate outcomes, $\theta$, based on parameters, $\beta$. The relationship $\theta=\Theta(\beta)$ is stochastic and analytically intractable, but I ...
sheß's user avatar
  • 367
4 votes
1 answer
512 views

Expectation Maximisation vs Expectation Propagation in the context of Bayesian Networks

I am confused about Expectation Maximisation and Expectation Propagation algorithms in the context of Bayesian Networks, especially whether one comprise another. What is the difference between ...
Pumpkin's user avatar
  • 141
1 vote
1 answer
666 views

What is the difference between approximate bayesian computation vs approximate bayesian inference?

What are the main differences between approximate bayesian computation vs approximate bayesian inference? Are they essentially the same? Do they refer to the same of different family of models? My ...
Kirk Walla's user avatar
0 votes
1 answer
37 views

How to combine sampled data from the same population?

Let's say I have a friend and we both asked one group of people a different question. For example, I ask the group how old they are, and my friend asks them how much they weigh. If I meet up with my ...
StatsNoob007's user avatar
2 votes
0 answers
227 views

Computing gradient of KL-divergence

Consider a normal distribution $\mathcal N(\boldsymbol{\mu}(w), \boldsymbol{\Sigma}(w))$, with mean $\boldsymbol{\mu}(w)$ and covariance $\boldsymbol{\Sigma}(w)$ that are parameterized by a vector of ...
evo99's user avatar
  • 21
4 votes
2 answers
559 views

Variational inference with discrete variational parameters

Typically Variational Inference relies on taking gradient steps on KL divergence between the variational and true posterior, or on the ELBO. This does not seem valid when variational parameters are ...
Dion's user avatar
  • 954
2 votes
1 answer
229 views

Variational Inference with intractable score function

Is it possible to do ELBO maximization using stochastic gradient estimates (i.e. iteratively apply variational updates using (3) in http://proceedings.mlr.press/v33/ranganath14.pdf), when it's cheap ...
Dion's user avatar
  • 954
5 votes
1 answer
186 views

Variational inference with deterministic dependencies between variables

Suppose I have a probabilistic graphical model shown in the picture, in which all variables are binary, $c_1$ and $c_2$ are observed, and I want to use mean-field variational inference to estimate ...
Ruben van Bergen's user avatar
0 votes
0 answers
421 views

Approximating expectation with Taylor series

I want to get the second-order Taylor approximation for an expectation. I have the following distribution, which is a Generalized Dirichlet distribution with parameters $\boldsymbol\alpha$ and $\...
c.uent's user avatar
  • 115
9 votes
1 answer
534 views

Rao-Blackwellization in variational inference

The Black box VI paper introduces Rao-Blackwellization as a method to reduce the variance of the gradient estimator using score function, in section 3.1. However I don't quite get the basic idea ...
avocado's user avatar
  • 3,623
2 votes
1 answer
255 views

Variational Inference - deriving coordinate update equations for mixture model

I'm currently going through this paper by Blei et. al. that describes the setup and derivation of the coordinate ascent equations for a Gaussian mixture model with K components. I am having some ...
snickerdoodles777's user avatar
5 votes
1 answer
285 views

variational inference derivation

According to this lecture note, Eq. 25 gives the coordinate ascent update for latent variable $z_k$ as follows $$q^*(z_k)\propto\exp(E_{-k}[\log{p(z_k,Z_{-k},x)}])$$ and I understand the derivation ...
avocado's user avatar
  • 3,623
4 votes
1 answer
349 views

Questions about approximate inference and calculating the posterior predictive

As I understand, computing the exact posterior of parameters $p(\theta|x)$ is nearly always impossible since we need to compute the evidence $\sum_\theta p(x|\theta)p(\theta)$ with every possible ...
groove's user avatar
  • 503
4 votes
2 answers
420 views

Variational Inference: Ising Model

I am self learning Variational Inference. Currently I am reading the chapter 21 book from Murphy 1 and trying to understand the Ising model (21.3.2). The Ising model here is used as denoising ...
mgbacher's user avatar
  • 121
1 vote
0 answers
232 views

Efficient approximate marginal inference in variational auto-encoder

In Auto-Encoding Variational Bayes, authors mentioned that they proposed a solution to "Efficient approximate marginal inference of the variable $x$". I read through the paper and appendix, now ...
LKS's user avatar
  • 212
1 vote
0 answers
99 views

Jensen's inequality in Collaborative Topic Regression

I am reading the article Collaborative Topic Modeling for Recommending Scientific Articles and could notice the application of Jensen's inequality in order to define a bound from which optimization is ...
ewerlopes's user avatar
  • 173
4 votes
1 answer
133 views

How Can I teach someone "sampling from a given distribution" is hard?

For many people I know, they do not think sampling from a given distribution is a hard problem in general. For example, many software provide functions do to sample from normal distribution or uniform ...
Haitao Du's user avatar
  • 37.1k
2 votes
1 answer
481 views

Variational Inference of Univariate Gaussian mixtures

I am reading this paper. In the paper, they use an example of mixture of unit-variance univariate Gaussians with the following parameterization: \begin{align} \mu_k & \sim \mathcal{N}(0, \sigma^2)...
Zhiya's user avatar
  • 241
9 votes
2 answers
6k views

Gradient of the expectation of a function w.r.t. distribution parameters

In section 2.2 of Kingma & Welling's paper on variational auto-encoders authors write the following equality for the gradient of the expectation of a function with respect to the parameters of the ...
ted's user avatar
  • 741
5 votes
2 answers
3k views

Difference between stochastic variational inference and variational inference?

Very simple, as the question header says: what is the difference between SVI and VI? I cannot seem to find a clear-cut definition online.
Astrid's user avatar
  • 989
2 votes
1 answer
853 views

Why does detailed balance not provide a stopping criterion in MCMC?

Like I undestand MCMC sampling, the fulfillment of the detailed balance equation guarantees that our MC has reached its stationary distribution (given we ensure ergodicity). Detailed Balance is: $\...
dopexxx's user avatar
  • 141
2 votes
1 answer
326 views

Normalizing Flows, Real NVPs and Inverse Autoregressive Flows - Used for Probabilty Density Approximation or for Sampling?

Suppose we have a parametric family $g(x;\theta)$, where $\theta$ are the parameters. As far as I can tell, there are two ways we can use this family to model a probability distribution: Probability ...
Lior's user avatar
  • 567
4 votes
0 answers
528 views

Convergence of approximate Gibbs sampling

We have a bivariate random variable $(X,Y)$ for which sampling is challenging. If we were to know how to sample from the conditionals $(X|Y)$ and $(Y|X)$, we could get samples from the joint using ...
PAM's user avatar
  • 311
2 votes
2 answers
4k views

get probabilities from kernel density estimation pdf

I have data points located at $\mathbf{x}_i$ and I would like to a find quick and dirty way to calculate their probability of occurring (not the pdf) using kernel density estimation. Formally speaking,...
user3433489's user avatar
3 votes
1 answer
275 views

Bethe approximation for factor graphs

I am confused at computing Bethe approximation for factor graphs in here. It generalizes Bethe approxmiation in a pairwise case. However, I am wondering why (75) goes to (78) with (76): We can verify ...
Mou's user avatar
  • 678
2 votes
1 answer
295 views

How to compute the Gibbs free energy in Bethe approximation for MRF

Hi, I am learning loopy belief propagation for MRF. The general roadmap is to define a Bethe approximation, which is exact for a tree but wrong for general graphs. I'm currently stuck at the point to ...
Mou's user avatar
  • 678
3 votes
0 answers
76 views

Simple approximation of joint posterior

Consider the (hierarchical) Bayesian inference problem with two unknowns $(x,\theta)$ and data $y$. I'm using a very simple ("independence"?) approximation $$ p(x,\theta|y) \approx p(x|\theta_\star,y) ...
Patrick's user avatar
  • 852
9 votes
2 answers
3k views

Estimating the gradient of log density given samples

I am interested in estimating the gradient of the log probability distribution $\nabla\log p(x)$ when $p(x)$ is not analytically available but is only accessed via samples $x_i \sim p(x)$. There ...
jkt's user avatar
  • 563