Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [gibbs]

The Gibbs sampler is a simple form of Markov Chain Monte Carlo simulation, widely used in Bayesian statistics, based on sampling from full conditional distributions for each variable or group of variables. The name comes from the method being first used on Gibbs random fields modeling of images by Geman and Geman (1984).

90
votes
3answers
52k views

Can someone explain Gibbs sampling in very simple words? [duplicate]

I'm doing some reading on topic modeling (with Latent Dirichlet Allocation) which makes use of Gibbs sampling. As a newbie in statistics―well, I know things like binomials, multinomials, priors, etc.―,...
3
votes
1answer
74 views

Sampling posterior of empty cluster in GMM and Gibbs

Consider performing inference via a standard Gibbs sampler for a standard Gaussian Mixture Model (GMM) with $k$ components that are Gaussians $$\mathcal{N}(\mu_{k}, \sigma^{2}_{k})$$ where we assume ...
3
votes
1answer
143 views

Importance weight of conditioned particle in conditional SMC

In a generic particle filter, I understand the importance weights for each particle are calculated as $w_t^s \propto w_{t-1}^s \frac{p(y_t \mid z_t^s) p(z_t^s \mid z_{t-1}^s)}{q(z_t^s \mid z_{t-1}^s, ...
15
votes
1answer
1k views

Stan $\hat{R}$ versus Gelman-Rubin $\hat{R}$ definition

I was going through the Stan documentation which can be downloaded from here. I was particularly interested in their implementation of the Gelman-Rubin diagnostic. The original paper Gelman & ...
5
votes
1answer
11k views

Gibbs sampler examples in R [closed]

How can I implement Gibbs sampler for the posterior distribution, and estimating the marginal posterior distribution by making histogram?
0
votes
1answer
40 views

Distribution sampling when no analytic expression

The goal of gibbs sampling is to sample the joint distribution when this latter has not an analytic expression, by deriving the conditional distribution of each variable. So it is supposed that the ...
4
votes
2answers
374 views

Posterior computation for Laplace distribution

I am dealing with being Bayesian and looking for a closed form for a posterior for the scale parameter $\tau$ of a Laplace distribution, such that I can derive a full conditional in my Gibbs sampler. ...
2
votes
2answers
68 views

Generating samples from Gibbs method

I have a following homework in a subject called "Monte-Carlo Methods". I would be very thankful, if you could help me with this one, because I'm a bit stuck with this one .. The task is as follows:...
5
votes
1answer
2k views

Sampling from an Inverse Gamma distribution

I am using Gibbs sampling in the MCMC estimation of a stochastic volatility model. One of the posterior distributions is an Inverse Gamma distribution.I was struggling with the sampling procedure or ...
3
votes
1answer
78 views

Gibbs Sampling vs. Using Raw Probability in Contrastive Divergence

In Hinton's Practical Guide to Training Restricted Boltzmann Machines, Section 3, he discusses different situations in which one should take a sample from the Gibbs sampling process, and other ...
2
votes
0answers
125 views

Bayesian prior and posterior computation for a truncated normal

I have to deal with data in a Bayesian framework, ultimately devising a Gibbs sampler for inferring all my distributions parameters. Specifically, suppose I observe some univariate data distributed ...
1
vote
0answers
50 views

Non-Identifiable Multivariate Normal Posterior

So I have a theoretical question about what looks like, in my opinion, a multivariate normal distribution. The issue comes with the fact that the data is distributed with likelihood: Y |θ1, θ2 ∼ N(θ1 ...
2
votes
1answer
157 views

Why can Gibbs sampling outputs be used in Rao-Blackwellization?

I'm currently learning Chib (1995)'s method to calculate the marginal likelihood of a Bayesian model using Gibbs sampling outputs. I'm stuck in the Rao-Blackwellization step. Suppose $\mu$ and $\phi$...
11
votes
1answer
3k views

Gibbs sampling for Ising model

Homework question: Consider the 1-d Ising model. Let $x = (x_1,...x_d)$. $x_i$ is either -1 or +1 $\pi(x) \propto e^{\sum_{i=1}^{39}x_ix_{i+1}}$ Design a gibbs sampling algorithm to generate ...
3
votes
0answers
181 views

Is burn-in necessary for MCMC/Gibbs sampling if I have samples from the true distribution already?

Say I have some samples from a distribution $p$, and I want to get more samples using MCMC/Gibbs sampling. Since the existing samples are known from the equilibrium distribution $p$, if I use them as ...
2
votes
1answer
71 views

Why does $P(\theta_1\mid D, \theta_2) \propto P(D \mid \theta_1, \theta_2)P(\theta_1)$ hold?

Suppose that in a Bayesian framework we have observed data $D$, using independent prior distributions on the parameters of the model, denoted by $\theta_1, \theta_2$. Then, the joint posterior ...
1
vote
0answers
218 views

plot log likelihood function evolution in mcmc simulations

Is it possible to plot log likelihood function evolution in mcmc simulations? I have a mixture model and its parameters are estimated using the gibbs sampling method in r environment and using the ...
0
votes
1answer
101 views

prior for initial values of Kalman Filter

I'm studying Carter and Kohn's (1994) implementation of the Gibbs sampler for Bayesian analysis of state space models. In their paper, they assume the starting value, call it $\beta_0$, of the state ...
1
vote
1answer
299 views

When can the collapsed Gibbs sampler be implemented?

I understand Gibbs sampling is a means of statistics inference, and it seems that sometimes certain variables can be integrated out in the sampling process, known as collapsed Gibbs sampling. I really ...
5
votes
1answer
79 views

How to use Gibbs sampling when target function is known only up to normalising constant?

Assume we want to use Gibbs sampling to get an estimate of the parameter , and that we have the following expression for the conditional posterior of : If I am not mistaken, this means that the ...
3
votes
2answers
273 views

Gibbs sampling and Bayesian inference

I wanted to get a more in-depth understanding of sampling algorithms, and so I thought I could start with the very simple example of a binomial or bernoulli likelihood with a beta prior (since it is ...
1
vote
0answers
54 views

help interpreting plot of MCMC sample

I am estimating a model using MCMC (Gibbs Sampling). Because of the complexity of the model, I have been running two chains with many iterations. A plot of the draws for each parameter reveals a ...
1
vote
0answers
69 views

Burning and Sampling in Gibbs sampler [closed]

I am confused about the concept of sampling in the Gibbs sampler after the burn-in loops. This is the basic problem, I have a picture composed of 1's and 0's. This is a noisy version and I am trying ...
11
votes
1answer
467 views

How to test if a cross-covariance matrix is non-zero?

The background of my study: In a Gibbs sampling where we sample $X$ (the variable of interests) and $Y$ from $P(X|Y)$ and $P(Y|X)$ respectively, where $X$ and $Y$ are $k$-dimensional random vectors. ...
0
votes
2answers
164 views

How is the accuracy of the results of Gibbs sampler measured?

How is the accuracy of the results of Gibbs sampler measured? Most resources merely say to iterate it $k$ times. But how does one infer the accuracy of the result?
1
vote
0answers
81 views

How to know whether a Gibbs sampler is irreducible? [duplicate]

How to know whether a Gibbs sampler is irreducible? I know that the Gibbs sampler in e.g. two variable case constructs a sequence of r.v.s $(X_1^{(i)}, X_2^{(i)})$ by sampling from the related ...
6
votes
1answer
1k views

Metropolis-Within-Gibbs sampling with only marginal distribution known for a subset of variables

Typically in Gibbs sampling we want to sample from a joint distribution $p(X_1, X_2, ..., X_N)$, but because the joint is hard to sample from directly, we instead achieve this by iteratively sampling ...
2
votes
0answers
179 views

Missing data in Gibbs sampling for dynamic linear models

Suppose I have the following DLM: $x_t = \Phi x_{t-1} + w_t$ $y_t = A x_t + v_t$ $x_0 \sim N(\mu_0,\Sigma_0)$ $w_t \sim N(0,Q)$ $v_t \sim N(0,R)$ Let $\Theta = \{\mu_0,\Sigma_0,\Phi,Q,A,R\}$. I ...
12
votes
1answer
290 views

Why does the redundant mean parameterization speed up Gibbs MCMC?

In Gelman & Hill (2007)'s book (Data Analysis Using Regression and Multilevel/Hierarchical Models), the authors claim that including redundant mean parameters can help speed up MCMC. The given ...
34
votes
1answer
12k views

What is the difference between Metropolis Hastings, Gibbs, Importance, and Rejection sampling?

I have been trying to learn MCMC methods and have come across Metropolis Hastings, Gibbs, Importance, and Rejection sampling. While some of these differences are obvious, i.e., how Gibbs is a special ...
4
votes
1answer
1k views

Explanation regarding Gibbs Sampling

I am new to MCMC and reading a intro paper regarding Gibbs sampling. However, there are two parts in the paper I cannot understand and get stuck. The first part is equation 2.3 in page 168. It says ...
1
vote
0answers
19 views

Averaging Across Gibbs Sampling Runs with Reduced Dimensions

I need help thinking through my approach to Gibbs Sampling of many parameters and I'd like to know if there is literature on this topic: I have a dataset with 3 dimensions: ...
3
votes
0answers
243 views

Gibbs sampling for parameter estimation

I am reading the paper by Willemsen et al (2015), "A multivariate Bayesian model for embryonic growth", Statistics in Medicine, 34:8, 1351–1365 where they define the posterior distribution as, \begin{...
2
votes
1answer
337 views

Why does detailed balance not provide a stopping criterion in MCMC?

Like I undestand MCMC sampling, the fulfillment of the detailed balance equation guarantees that our MC has reached its stationary distribution (given we ensure ergodicity). Detailed Balance is: $\...
1
vote
1answer
540 views

deriving posterior conditionals for gibbs sampling

I'm new to Bayesian inference and Gibbs sampling in general, and I'm struggling trying to derive the conditional posteriors for a particular data generating process I'm trying to model. The model I ...
2
votes
1answer
66 views

Generating samples for $p(\theta_{i}|\pmb{x})$ if samples from $p(\phi|\pmb{x})$ are known

Suppose $X_{i}|\theta_{i} \sim D_{1}(\theta_{i})$ and $\theta_{i}|\phi \sim D_{2}(\phi)$. Moreover $\phi \sim D_{3}(c)$ where c is known. How would I generate samples for $p(\theta_{i}|\pmb{x})$ if I ...
3
votes
0answers
269 views

Convergence of approximate Gibbs sampling

We have a bivariate random variable $(X,Y)$ for which sampling is challenging. If we were to know how to sample from the conditionals $(X|Y)$ and $(Y|X)$, we could get samples from the joint using ...
2
votes
1answer
290 views

Gibbs sampling version for estimating the Dynamic Topic Model (DTM)?

The paper of Blei et Lafferty published at ICML'06 implements a (quite complicated) variational inference (VI) technique for estimating the parameters of the Dynamic Topic Model, see: http://pdf....
0
votes
1answer
210 views

Gibbs sampler transition kernel proof [duplicate]

I'm trying to understand how the Gibbs sampling algorithm works. I've simplified it into a bivariate case to help my understanding but I'm unsure how to go from conditioning on $X^{t-1},Y^{t-1},X^t$ ...
0
votes
1answer
96 views

Sampling from conditional posterior - continuous and discrete terms

I'm working through Hierarchically Supervised LDA by Perotte et all (2011). The conditional posterior I'm supposed to sample values $z_i$ from, however, is zero almost everywhere. To see why, lets ...
41
votes
4answers
6k views

OpenBugs vs. JAGS

I am about to try out a BUGS style environment for estimating Bayesian models. Are there any important advantages to consider in choosing between OpenBugs or JAGS? Is one likely to replace the other ...
2
votes
1answer
1k views

Gibbs measure and normal distribution

On Wikipedia, the Gibbs measure defines the probability as: $$ P(X=x) = \frac{1}{Z(\beta)}\exp(-\beta E(x)) $$ Now, the familiar form of the normal distribution is: $$ P(x) = \frac{1}{\sqrt{2\pi}\...
1
vote
1answer
185 views

Sampling from a posterior with Gibbs sampling

In an image processing class, I dont really get behind the idea how to 'sample from a posterior' with Gibbs sampling. We have a posterior distribution: $f(z_1, .. ,z_n \mid x_1,.. ,x_n) := f(z \mid x)...
3
votes
1answer
558 views

Gibbs sampling an Ising model with 0s and 1s

One of my problems in one of my courses ask to sample a 20 dimensional vector of 0s and 1s, $\{0,1\}^{20},$ when they are distributed as $$ \pi(x) = \exp\left\{-\beta \sum_{i=1}^{19} |x_{i+1}-x_i| \...
1
vote
0answers
29 views

Gibbs sampling with expectations instead of sampling

I see there is something called Iterated Conditional Models (ICM), which is a sort of Gibbs sampling where, instead of sampling, we use the value that maximizes the conditional. That is: ...
3
votes
1answer
466 views

Blocked Gibbs Sampling using Forward / Backward Algorithm

I am new to machine learning and have been reading about gibbs sampling. From my understanding, a Gibbs algorithm samples a single variable iteratively conditioned on all other variables. In blocked ...
0
votes
1answer
53 views

Priors on Taylor Expansion series

I'm wondering what priors can i choose for a Taylor series as follows: $\theta_{1}+\theta_{2} (y-\alpha) + \theta_{3} (y-\alpha)^2$ What priors should I use for updating these parameters ($\theta_{1},...
1
vote
0answers
221 views

Collapsed Gibbs sampler on Hierarchical Dirichlet Process Mixture Model

I am trying to design a collapsed Gibbs sampler on a mixture model based on Hierarchical Dirichlet Process ($g\sim DP(\gamma, b)$, $\pi\sim DP(\alpha, g)$ ). Should I resample from the posterior of ...
0
votes
1answer
112 views

Rewrite conditional formula with three variables using Bayes formula

In equation (5) on page 3 on this paper a conditional probability is rewritten using Bayes' formula. I started using this answer Can I rewrite conditional probability of three variables like this? ...
4
votes
1answer
72 views

Sampling from multivariate normal conditional on a negative minimum

Let $X\sim \mathcal{N}(\mu,\Sigma)$, where $\mu\in\mathbb{R}^n$ and $\Sigma\in\mathbb{R}^{n\times n}$. How can I efficiently sample from $X | {\min{X}\le 0}$? (I.e. from the distribution of $X$ ...