Questions tagged [gibbs]

The Gibbs sampler is a simple form of Markov Chain Monte Carlo simulation, widely used in Bayesian statistics, based on sampling from full conditional distributions for each variable or group of variables. The name comes from the method being first used on Gibbs random fields modeling of images by Geman and Geman (1984).

Filter by
Sorted by
Tagged with
91
votes
3answers
53k views

Can someone explain Gibbs sampling in very simple words? [duplicate]

I'm doing some reading on topic modeling (with Latent Dirichlet Allocation) which makes use of Gibbs sampling. As a newbie in statistics―well, I know things like binomials, multinomials, priors, etc.―,...
41
votes
4answers
6k views

OpenBugs vs. JAGS

I am about to try out a BUGS style environment for estimating Bayesian models. Are there any important advantages to consider in choosing between OpenBugs or JAGS? Is one likely to replace the other ...
35
votes
1answer
12k views

What is the difference between Metropolis Hastings, Gibbs, Importance, and Rejection sampling?

I have been trying to learn MCMC methods and have come across Metropolis Hastings, Gibbs, Importance, and Rejection sampling. While some of these differences are obvious, i.e., how Gibbs is a special ...
29
votes
3answers
10k views

A good Gibbs sampling tutorials and references

I want to learn how Gibbs Sampling works and I am looking for a good basic to intermediate paper. I have a computer science background and basic statistic knowledge. Anyone has read good material ...
21
votes
1answer
1k views

What are some well known improvements over textbook MCMC algorithms that people use for bayesian inference?

When I'm coding a Monte Carlo simulation for some problem, and the model is simple enough, I use a very basic textbook Gibbs sampling. When it's not possible to use Gibbs sampling, I code the textbook ...
20
votes
2answers
8k views

Gibbs sampling versus general MH-MCMC

I have just been doing some reading on Gibbs sampling and Metropolis Hastings algorithm and have a couple of questions. As I understand it, in the case of Gibbs sampling, if we have a large ...
16
votes
1answer
8k views

When would one use Gibbs sampling instead of Metropolis-Hastings?

There are different kinds of MCMC algorithms: Metropolis-Hastings Gibbs Importance/rejection sampling (related). Why would one use Gibbs sampling instead of Metropolis-Hastings? I suspect there ...
16
votes
1answer
3k views

Does the Gibbs Sampling algorithm guarantee detailed balance?

I have it on supreme authority1 that Gibbs Sampling is a special case of the Metropolis-Hastings algorithm for Markov Chain Monte Carlo sampling. The MH algorithm always gives a transition probability ...
15
votes
2answers
7k views

Where do the full conditionals come from in Gibbs sampling?

MCMC algorithms like Metropolis-Hastings and Gibbs sampling are ways of sampling from the joint posterior distributions. I think I understand and can implement metropolis-hasting pretty easily--you ...
15
votes
1answer
1k views

Stan $\hat{R}$ versus Gelman-Rubin $\hat{R}$ definition

I was going through the Stan documentation which can be downloaded from here. I was particularly interested in their implementation of the Gelman-Rubin diagnostic. The original paper Gelman & ...
12
votes
1answer
1k views

Marginal Likelihood from the Gibbs Output

I'm reproducing from scratch the results in Section 4.2.1 of Marginal Likelihood from the Gibbs Output Siddhartha Chib Journal of the American Statistical Association, Vol. 90, No. 432. (Dec., 1995)...
12
votes
1answer
296 views

Why does the redundant mean parameterization speed up Gibbs MCMC?

In Gelman & Hill (2007)'s book (Data Analysis Using Regression and Multilevel/Hierarchical Models), the authors claim that including redundant mean parameters can help speed up MCMC. The given ...
11
votes
2answers
690 views

Is Gibbs sampling an MCMC method?

As far as I understand it, it is (at least, that is how Wikipedia defines it). But I've found this statement by Efron* (emphasis added): Markov chain Monte Carlo (MCMC) is the great success story ...
11
votes
1answer
3k views

Gibbs sampling for Ising model

Homework question: Consider the 1-d Ising model. Let $x = (x_1,...x_d)$. $x_i$ is either -1 or +1 $\pi(x) \propto e^{\sum_{i=1}^{39}x_ix_{i+1}}$ Design a gibbs sampling algorithm to generate ...
11
votes
1answer
2k views

How to derive Gibbs sampling?

I'm actually hesitating to ask this, because I'm afraid I will be referred to other questions or Wikipedia on Gibbs sampling, but I don't have the feeling that they describe what's at hand. Given a ...
11
votes
2answers
1k views

How do programs like BUGS/JAGS automatically determine conditional distributions for Gibbs sampling?

Seems like full conditionals are often quite difficult to derive, yet programs like JAGS and BUGS derive them automatically. Can someone explain how they algorithmically generate full conditionals for ...
11
votes
1answer
471 views

How to test if a cross-covariance matrix is non-zero?

The background of my study: In a Gibbs sampling where we sample $X$ (the variable of interests) and $Y$ from $P(X|Y)$ and $P(Y|X)$ respectively, where $X$ and $Y$ are $k$-dimensional random vectors. ...
10
votes
1answer
550 views

Bayesian modeling using multivariate normal with covariate

Suppose you have an explanatory variable ${\bf{X}} = \left(X(s_{1}),\ldots,X(s_{n})\right)$ where $s$ represents a given coordinate. You also have a response variable ${\bf{Y}} = \left(Y(s_{1}),\ldots,...
9
votes
2answers
2k views

Confusion related to Gibbs sampling

I came across this article where it says that in Gibbs sampling every sample is accepted. I am a bit confused. How come if every sample it accepted it converges to a stationary distribution. In ...
8
votes
2answers
2k views

Rao-Blackwellization of Gibbs Sampler

I am currently estimating a stochastic volatility model with Markov Chain Monte Carlo methods. Thereby, I am implementing Gibbs and Metropolis sampling methods.Assuming I take the mean of the ...
8
votes
1answer
634 views

Gibbs Sampler transition kernel

Let $\pi$ be the target distribution on $(\mathbb{R}^d,\mathcal{B}(\mathbb{R^d}))$ which is absolutely continuously wrt to the $d$-dimensional Lebesgue measure, i.e : $\pi$ admits a density $\pi(x_1,....
8
votes
1answer
147 views

Would an “importance Gibbs” sampling method work?

I suspect this is a fairly unusual and exploratory question, so please bear with me. I am wondering if one could apply the idea of importance sampling to Gibbs sampling. Here's what I mean: in Gibbs ...
8
votes
1answer
3k views

Bayesian estimation of Dirichlet distribution parameters

I want to estimate parameters of Dirichlet mixture models using Gibbs sampling and I have some questions about that: Is a mixture of Dirichlet distributions equivalent to a Dirichlet process? What is ...
8
votes
1answer
722 views

Can I subsample a large dataset at every MCMC iteration?

Problem: I want to perform a Gibbs sampling to infer some posterior over a large dataset. Unfortunatelly, my model is not very simple and thus sampling is too slow. I would consider variational or ...
8
votes
0answers
59 views

What is the correct way to write the elastic net?

I am confused about the correct way to write the elastic net. After reading some research papers there seems to be three forms 1) $\exp\{-\lambda_1|\beta_k|-\lambda_2\beta_k^2\}$ 2) $\exp\{-\frac{(\...
7
votes
2answers
8k views

Generating samples from Gibbs sampling

I am quite new to sampling. I am doing Gibbs sampling for a Bayesian network. I am aware about the algorithm for the Gibbs sampling but there's one thing I am not able to understand. For example let'...
7
votes
3answers
5k views

PyMC: how can I define a function of two stochastic variables, with no closed-form distribution?

I'm learning PyMC and basically I have a random variable $Z = X + Y$ where (say) $X \sim \mathrm{Normal}(\theta_X)$ and $Y \sim \mathrm{Lognormal}(\theta_Y)$ and $Z$ has no simple closed-form ...
7
votes
2answers
2k views

Gibbs sampler from conditional distribution

I am trying to propose Gibbs sampling with the density below, $$p(y_1,y_2,y_3)\propto \exp [-({{y}_{1}}+{{y}_{2}}+{{y}_{3}}+{{\theta}_{12}{y_1}{y_2}}+{{\theta }_{13}{y_3}{y_1}}+{{\theta }_{23}{y_2}{...
7
votes
3answers
2k views

Why does sampling from the posterior predictive distribution $p(x_{new} \mid x_1, \ldots x_n)$ work without having to average out the integral?

In a Bayesian model, the posterior predictive distribution is usually written as: $$ p(x_{new} \mid x_1, \ldots x_n) = \int_{-\infty}^{\infty} p(x_{new}\mid \mu) \ p(\mu \mid x_1, \ldots x_n)d\mu $$ ...
7
votes
1answer
3k views

Gibbs sampling to produce posterior pdf

Suppose we have the following classical normal linear regression model: $$y_i = \beta_1 x_{1i} + \beta_2x_{2i} + \beta_3x_{3i} + e_i$$ where $e_{i} \sim iid.N(0, \sigma^2)$ for all $i = 1, 2, \cdots,...
7
votes
1answer
177 views

Gibbs Sampler output: how many Markov chains?

When running a Gibbs sampler (for $n=200$ Iterations) with two full conditionals, I get the output $\mathbf{x} = (x_1^{(n)},x_2^{(n)})_{n =1,...,200}$. So $\mathbf{x}$ is the realizations of a Gibbs ...
7
votes
1answer
224 views

Given that one can sample $X \sim f(x)$, is there an easy way to sample $Y \sim k \cdot f(g(y))$ (such as $k \cdot f(e^y)$)?

Say I'm able to sample an RV $X$ from a PDF $f(x)$, can I exploit this to efficiently sample another RV $Y \sim k \cdot f(g(y))$ (where $k$ is a normalizing constant)? I'm interested in something ...
6
votes
1answer
11k views

Gibbs sampler examples in R [closed]

How can I implement Gibbs sampler for the posterior distribution, and estimating the marginal posterior distribution by making histogram?
6
votes
2answers
5k views

Gibbs sampling how to sample from the conditional probability? Bayesian model

I want to learn Gibbs sampling for a Bayesian model. How can I sample the variable from the conditional distribution? In this example, arrow means dependent; for example, ...
6
votes
2answers
1k views

Gibbs sampler gets stuck in local mode

I am very new to statistics and trying to implement a Gibbs sampler. However, according to wikipedia https://en.wikipedia.org/wiki/Gibbs_sampling and this discussion thread http://metaoptimize.com/qa/...
6
votes
1answer
1k views

Zero-inflated Poisson and Gibbs sampling, proofs and sampling

I am trying to figure out zip-inflated Poisson (ZIP) model. In this model, random data $X_1, .., X_n$ are of the form $X_i=R_iY_i$, where the $Y_i$'s have Poisson distribution ($\lambda$) and the $R_i$...
6
votes
1answer
2k views

Metropolis-Within-Gibbs sampling with only marginal distribution known for a subset of variables

Typically in Gibbs sampling we want to sample from a joint distribution $p(X_1, X_2, ..., X_N)$, but because the joint is hard to sample from directly, we instead achieve this by iteratively sampling ...
6
votes
1answer
281 views

Gibbs Sampler contradiction proof

I want to prove that the systematic scan Gibbs sampler yields an aperiodic chain $X$ on a general state space. Let $\pi$ be the stationary distribution for the resulting chain. Suppose to get a ...
6
votes
1answer
190 views

Dirichlet process mixture MCMC

I'm reading Markov Chain Sampling Methods for Dirichlet Process Mixture Models by Radford M. Neal. Equation (3.6) states that $$ \text{If } c=c_{j} \text{ for some } j\neq i: P\left(c_{i}=c\;|\;c_{-i}...
6
votes
0answers
740 views

Is my OpenBUGS / WinBUGS model well specified?

I've just started trying to use OpenBUGS for Bayesian analysis of stochastic volatility models. In particular, I'm trying to calculate stochastic covariance, similar to the DC-MSV model specified by ...
5
votes
3answers
243 views

Conditional distribution of $\exp(-|x|-|y|-a \cdot |x-y|)$

I am trying to use Gibbs sampling or Metropolis-Hastings to draw samples from the joint distribution$$f(x,y)\propto\exp(-|x|-|y|-a \cdot |x-y|)$$ For this I need the conditional distributions of $x$ ...
5
votes
1answer
2k views

Sampling from an Inverse Gamma distribution

I am using Gibbs sampling in the MCMC estimation of a stochastic volatility model. One of the posterior distributions is an Inverse Gamma distribution.I was struggling with the sampling procedure or ...
5
votes
1answer
2k views

Preparing Bayesian conditional distributions for Gibbs sampling

I was looking at the Gibbs Sampler when I stumbled upon the following example: Suppose $y = (y_{1}, y_{2}, \ldots, y_{n})$ are iid observations from an $N(\mu, \tau^{-1})$ Furthermore, suppose there ...
5
votes
1answer
2k views

Bayesian regression full conditional distribution

I have a problem with the derivation of the full conditional distribution of the regression coefficients in a simple Bayesian regression. The source of the following equations is: Lynch (2007). ...
5
votes
1answer
867 views

Gibbs sampling from a complex full conditional

I have a sampling question relating to Gibbs sampling of a complicated full conditional. Supposed I have a complicated full conditional that I want a single sample from $p(\theta_i$|$\theta_{-i}$, $...
5
votes
1answer
308 views

Why does Slice sampler use the log of the density?

The Slice sampler1 takes as its argument the log of the density to be sampled from. Why is it doing this? A commenter on this question pointed out that it makes no sense to "sample" from the log of ...
5
votes
2answers
4k views

How to reduce autocorrelation in Metropolis algorithm?

I've been using a Metropolis/Gibbs sampler combination to generate a joint density for some parameters(it is a hierarchical model, with $y_i\sim Poisson(\lambda_i)$, $\lambda_i\sim Gamma(\alpha,\beta)...
5
votes
1answer
668 views

Confusion in Gibbs sampling

I am self-studying Gibbs sampling from a book. The book introduces metropolis hastings algortihm to generate representative values from a posterior distribution. So we know $p(D | \theta) p(\theta)$ ...
5
votes
2answers
281 views

Gibbs Sampling and Probability Notation

Problem 1 I am trying to implement Gibbs Sampling for the following problem: There is a grid measuring 3 x 3 sites, each "site" can be designated in a state, $X$, of 1 or -1. The sites are numbered ...
5
votes
1answer
1k views

Convergence results for block-gibbs sampling?

Suppose you have some complex model you want to sample from by Markov chain Monte Carlo. There are many types of situations where you can divide your variables into, say, two groups, and efficiently ...