Questions tagged [approximate-inference]
The approximate-inference tag has no usage guidance.
62
questions
0
votes
0
answers
33
views
Is the inferential challenge of dense bayesian or markov networks solved after current improvements in variational inference and neural networks?
I am trying to understand more about Graphical models, and have a reasonable grasp of the basics now.
One issue that recurs in a lot of the papers of the mid-2000s and even in Koller's textbook is ...
- 1,532
3
votes
2
answers
231
views
Why don’t diffusion models suffer posterior collapse?
In VAEs, posterior collapse occurs when the approximated posterior $q_\theta(z|x)$ becomes the standard Gaussian prior $p(z)$ after training (Lucas et al. 2019). The forward process of diffusion ...
- 31
1
vote
0
answers
23
views
How do I estimate probability of someone having condition X [closed]
There is a survey, which asks following question: do you know personally someone, who has condition X?
The result of the survey is percentage of people, who know someone
I want to use this survey to ...
- 111
1
vote
1
answer
95
views
Does approximating the likelihood function violate the likelihood principle in Bayesian Inference?
Suppose we have a prior $p(\theta)$ and a likelihood function $L(\theta|x)$, and that the likelihood $L(\theta|x)$ is intractable somehow (difficult or impossible to compute) and we instead replace it ...
- 11
1
vote
1
answer
30
views
A soft question about probabilistic inference
Suppose A and B are two propositions and $ A \implies B$ is true.WE know all it means is that B is true whenever A is true. Now consider a situation where we only that A is true with probability ,say,...
- 437
1
vote
1
answer
103
views
Generative model that satisfies certain algebraic constraints
Disclaimer: I need guidance and help with where to start looking for solution of the problem I have described below. My background is in optimization and I am new to statistical methods, so there is a ...
- 111
0
votes
0
answers
27
views
How to approximate the expression to $\sum x_i$
How to approximate the expression on the left hand side to $\sum_{i=1}^Nx_i$ as $n\to \infty$
$$ \frac{\sum\limits_{i=1}^{N}x_i^2}{n-2\frac{\sum\limits_{i=1}^{N}x_i}{N}}
\left(\sqrt{1+\frac{Nn\left(...
- 11
0
votes
1
answer
41
views
Regression where the matrix is drawn from a known distribution
I need to estimate the vector x, where Ax = b, given b is observed data and the columns in the matrix A are drawn from a known distribution. a and b are both formed of positive integers (or zero).
For ...
- 1
0
votes
2
answers
110
views
Conditional vs. nonconditional variational family
In variational bayes, distributions in the variational family $\mathcal{Q}$ are denoted $q_\phi$ and are used to approximate the posterior $p_\theta(z|x)$. However, I've seen both notations $q_\phi(z|...
- 399
3
votes
1
answer
1k
views
Understanding the set of latent variables $Z$ in variational inference
I have been trying to understand variational inference (in a Bayesian context) where we’re trying to approximate $p(Z|X)$ where $Z$ is the set of latent variables and $X$ is the set of observable ...
- 33
1
vote
0
answers
56
views
Non-submodularity of sparse approximations to Gaussian posterior
Imagine that we are doing approximate inference on a Gaussian mean, minimizing a variational objective that corresponds to the reverse KL divergence between a sparse approximate posterior computed on ...
- 964
0
votes
1
answer
28
views
Average sales of last three years using interpolation/extrapolation
I have average sales value for the last three years of a company, e.g.
year
avg. sales
2017
100
2018
150
2019
200
Is it possible to somehow come up with an estimate of total average sales value ...
- 31
1
vote
1
answer
887
views
Are VAE useful for Maximum-Likelihood estimation?
In the "Auto-encoding Variation Bayes" Paper they state under "2.1 Problem Scenarios" that the VAE is a solution to:
"1. Efficient approximate ML or MAP estimation for the ...
- 51
1
vote
0
answers
66
views
Multinomial distribution
I want to show this theorem. here you get the details of it. http://yaroslavvb.com/upload/wasserman-multinomial.pdf?fbclid=IwAR1V6yB5-100XSzk6vt5cgNzV0PHdT9qJ8mwxEGplV1q-QiwgP3-Z7i0jvA
where $p$ is ...
- 21
1
vote
1
answer
193
views
Introduction to approximate message passing
I'm interested in learning approximate message passing from the paper "Message Passing Algorithms for Compressed Sensing: I. Motivation and Construction". My background is in computer ...
- 117
1
vote
0
answers
30
views
Are there convenient methods/tricks to make calculations with non-independent terms? (two examples here in particular)
In a recent question it came out that I needed to calculate the sample distribution of $\dfrac{Cov(XY,X)}{Var(X)}$ where the distributions of $X$ and $Y$ are known.
By this, I mean the law of $$\dfrac{...
- 704
1
vote
1
answer
205
views
What is a Dirac distribution on a hyperplane?
I'm trying to understand message passing for compressed sensing. I came acrross this distribution:
As the title suggests, how does this distribution look like? I know the first products term in the ...
- 117
2
votes
0
answers
68
views
What is the link between the queries Bayesian Networks can answer, and inference algorithms?
I have seen two concepts linked to Bayesian Networks:
Bayesian Networks can answer different types of queries. These types include proof of evidence, most probable explanation, computing maximum a ...
1
vote
0
answers
38
views
Can we ignore the generation side of the method described in density estimation using Real NVP?
First appologies if my question is stupid. I am studying the paper "Density estimation using real NVP" by Dinh, Sohl-Dickstein and Bengio. link
The paper presented a nice idea that the generation ...
- 11
0
votes
1
answer
164
views
Difference / Relationship of Generative Models / Variational Bayesian Inference
I feel a bit confused trying to merge and unify understandings of generative models and variational bayesian inference methods. Initially, I believed them to be the same thing, namely learning full ...
user143877
2
votes
1
answer
143
views
Derivation of the Objective Function for Expectation Propagation
I was reading Expectation Propagation As A Way Of Life and the original paper by Minka Expectation Propagation for Approximate Bayesian Inference and they both say that a fixed point of the EP ...
- 2,246
0
votes
0
answers
40
views
Exact inference in an approximate model as opposed to approximate inference in an exact model?
I remember hearing a while ago that it was more rigorous to perform approximate inference in an exact model as opposed to exact inference in an approximate model. I can’t now remember the reasoning ...
- 21
1
vote
0
answers
11
views
Approximate Bayesian computation for comparing parameters affect on a response variable [closed]
I'm not sure I understand this method perfectly so please correct me if I'm wrong. From my understanding Approximate Bayesian Computation allows you to perform likelihood free inference by re-...
- 576
3
votes
1
answer
135
views
How to Test Linear Hypotheses about Parameters in Simulation-Based Indirect Inference
Setup:
I have a model that produces a vector of aggregate outcomes, $\theta$, based on parameters, $\beta$. The relationship $\theta=\Theta(\beta)$ is stochastic and analytically intractable, but I ...
- 367
4
votes
1
answer
534
views
Expectation Maximisation vs Expectation Propagation in the context of Bayesian Networks
I am confused about Expectation Maximisation and Expectation Propagation algorithms in the context of Bayesian Networks, especially whether one comprise another.
What is the difference between ...
- 141
1
vote
1
answer
690
views
What is the difference between approximate bayesian computation vs approximate bayesian inference?
What are the main differences between approximate bayesian computation vs approximate bayesian inference?
Are they essentially the same?
Do they refer to the same of different family of models?
My ...
- 212
0
votes
1
answer
38
views
How to combine sampled data from the same population?
Let's say I have a friend and we both asked one group of people a different question. For example, I ask the group how old they are, and my friend asks them how much they weigh. If I meet up with my ...
2
votes
0
answers
231
views
Computing gradient of KL-divergence
Consider a normal distribution $\mathcal N(\boldsymbol{\mu}(w), \boldsymbol{\Sigma}(w))$, with mean $\boldsymbol{\mu}(w)$ and covariance $\boldsymbol{\Sigma}(w)$ that are parameterized by a vector of ...
- 21
4
votes
2
answers
585
views
Variational inference with discrete variational parameters
Typically Variational Inference relies on taking gradient steps on KL divergence between the variational and true posterior, or on the ELBO. This does not seem valid when variational parameters are ...
- 964
2
votes
1
answer
239
views
Variational Inference with intractable score function
Is it possible to do ELBO maximization using stochastic gradient estimates (i.e. iteratively apply variational updates using (3) in http://proceedings.mlr.press/v33/ranganath14.pdf), when it's cheap ...
- 964
5
votes
1
answer
189
views
Variational inference with deterministic dependencies between variables
Suppose I have a probabilistic graphical model shown in the picture, in which all variables are binary, $c_1$ and $c_2$ are observed, and I want to use mean-field variational inference to estimate ...
- 7,702
0
votes
0
answers
426
views
Approximating expectation with Taylor series
I want to get the second-order Taylor approximation for an expectation.
I have the following distribution, which is a Generalized Dirichlet distribution with parameters $\boldsymbol\alpha$ and $\...
- 115
9
votes
1
answer
559
views
Rao-Blackwellization in variational inference
The Black box VI paper introduces Rao-Blackwellization as a method to reduce the variance of the gradient estimator using score function, in section 3.1.
However I don't quite get the basic idea ...
- 3,623
2
votes
1
answer
265
views
Variational Inference - deriving coordinate update equations for mixture model
I'm currently going through this paper by Blei et. al. that describes the setup and derivation of the coordinate ascent equations for a Gaussian mixture model with K components. I am having some ...
5
votes
1
answer
293
views
variational inference derivation
According to this lecture note, Eq. 25 gives the coordinate ascent update for latent variable $z_k$ as follows
$$q^*(z_k)\propto\exp(E_{-k}[\log{p(z_k,Z_{-k},x)}])$$
and I understand the derivation ...
- 3,623
4
votes
1
answer
357
views
Questions about approximate inference and calculating the posterior predictive
As I understand, computing the exact posterior of parameters $p(\theta|x)$ is nearly always impossible since we need to compute the evidence $\sum_\theta p(x|\theta)p(\theta)$ with every possible ...
- 503
4
votes
2
answers
426
views
Variational Inference: Ising Model
I am self learning Variational Inference. Currently I am reading the chapter 21 book from Murphy 1 and trying to understand the Ising model (21.3.2).
The Ising model here is used as denoising ...
- 141
1
vote
0
answers
237
views
Efficient approximate marginal inference in variational auto-encoder
In Auto-Encoding Variational Bayes, authors mentioned that they proposed a solution to "Efficient approximate marginal inference of the variable $x$". I read through the paper and appendix, now ...
- 212
1
vote
0
answers
100
views
Jensen's inequality in Collaborative Topic Regression
I am reading the article Collaborative Topic Modeling for Recommending Scientific Articles and could notice the application of Jensen's inequality in order to define a bound from which optimization is ...
- 173
4
votes
1
answer
135
views
How Can I teach someone "sampling from a given distribution" is hard?
For many people I know, they do not think sampling from a given distribution is a hard problem in general. For example, many software provide functions do to sample from normal distribution or uniform ...
- 37.3k
2
votes
1
answer
486
views
Variational Inference of Univariate Gaussian mixtures
I am reading this paper. In the paper, they use an example of mixture of unit-variance univariate Gaussians with the following parameterization:
\begin{align}
\mu_k & \sim \mathcal{N}(0, \sigma^2)...
- 241
9
votes
2
answers
6k
views
Gradient of the expectation of a function w.r.t. distribution parameters
In section 2.2 of Kingma & Welling's paper on variational auto-encoders authors write the following equality for the gradient of the expectation of a function with respect to the parameters of the ...
- 741
5
votes
2
answers
3k
views
Difference between stochastic variational inference and variational inference?
Very simple, as the question header says: what is the difference between SVI and VI?
I cannot seem to find a clear-cut definition online.
- 989
2
votes
1
answer
862
views
Why does detailed balance not provide a stopping criterion in MCMC?
Like I undestand MCMC sampling, the fulfillment of the detailed balance equation guarantees that our MC has reached its stationary distribution (given we ensure ergodicity).
Detailed Balance is:
$\...
- 141
2
votes
1
answer
331
views
Normalizing Flows, Real NVPs and Inverse Autoregressive Flows - Used for Probabilty Density Approximation or for Sampling?
Suppose we have a parametric family $g(x;\theta)$, where $\theta$ are the parameters. As far as I can tell, there are two ways we can use this family to model a probability distribution:
Probability ...
- 567
4
votes
0
answers
531
views
Convergence of approximate Gibbs sampling
We have a bivariate random variable $(X,Y)$ for which sampling is challenging.
If we were to know how to sample from the conditionals $(X|Y)$ and $(Y|X)$, we could get samples from the joint using ...
- 311
2
votes
2
answers
4k
views
get probabilities from kernel density estimation pdf
I have data points located at $\mathbf{x}_i$ and I would like to a find quick and dirty way to calculate their probability of occurring (not the pdf) using kernel density estimation. Formally speaking,...
- 423
3
votes
1
answer
284
views
Bethe approximation for factor graphs
I am confused at computing Bethe approximation for factor graphs in here. It generalizes Bethe approxmiation in a pairwise case. However, I am wondering why (75) goes to (78) with (76):
We can verify ...
- 678
2
votes
1
answer
298
views
How to compute the Gibbs free energy in Bethe approximation for MRF
Hi, I am learning loopy belief propagation for MRF. The general roadmap is to define a Bethe approximation, which is exact for a tree but wrong for general graphs.
I'm currently stuck at the point to ...
- 678
3
votes
0
answers
77
views
Simple approximation of joint posterior
Consider the (hierarchical) Bayesian inference problem with two unknowns $(x,\theta)$ and data $y$. I'm using a very simple ("independence"?) approximation
$$ p(x,\theta|y) \approx p(x|\theta_\star,y) ...
- 852