Skip to main content
39 votes
Accepted

When would one use Gibbs sampling instead of Metropolis-Hastings?

Firstly, let me note [somewhat pedantically] that There are several different kinds of MCMC algorithms: Metropolis-Hastings, Gibbs, importance/rejection sampling (related). importance and ...
Xi'an's user avatar
  • 106k
32 votes

Gibbs sampler examples in R

Problem Suppose $Y \sim \text{N}(\text{mean} = \mu, \text{Var} = \frac{1}{\tau})$. Based on a sample, obtain the posterior distributions of $\mu$ and $\tau$ using the Gibbs sampler. Notation $ \...
ocram's user avatar
  • 22k
15 votes
Accepted

Is Gibbs sampling an MCMC method?

The algorithm that is now called Gibbs sampling forms a Markov-chain and uses Monte-Carlo simulation for its inputs, so it does indeed fall within the proper scope of MCMC (Markov-Chain Monte-Carlo) ...
Ben's user avatar
  • 126k
14 votes

Can someone explain Gibbs sampling in very simple words?

I find this document GIBBS SAMPLING FOR THE UNINITIATED by Resnik & Hardisty very useful for non-statistics background folks. It explains why & how to use Gibbs sampling, and has examples ...
jfong's user avatar
  • 141
12 votes
Accepted

Rao-Blackwellization of Gibbs Sampler

Assuming I take the mean of the posterior distribution rather than a random sample from it, is this what is commonly referred to as Rao-Blackwellization? I am not very familiar with stochastic ...
Greenparker's user avatar
  • 15.6k
11 votes
Accepted

Posterior computation for Laplace distribution

The likelihood for $n$ iid observations looks like: $ f(x_1,...x_n|\lambda,\mu) \propto \frac{1}{\lambda^n} exp(-\frac{1}{\lambda}\sum_{i=1}^n|x_i-\mu|)$ Hence a conjugate prior for $\lambda$ with $\...
conjectures's user avatar
  • 4,236
11 votes
Accepted

Is the joint distribution $P_{XY}(x,y)$ determined from the conditionals $P_{X|Y}(x|y)$ and $P_{Y|X}(y|x)$?

This characterisation of the joint by the conditionals is the Hammersley-Clifford theorem. Unpublished by the authors but later established under the positivity condition by Julian Besag in 1974. The ...
Xi'an's user avatar
  • 106k
10 votes

Rao-Blackwellization of Gibbs Sampler

The Gibbs sampler can then be used to improve efficiency of (say) samples from a marginal posterior, call it $\pi_2(\theta_2|y)$. Note \begin{eqnarray*} \pi_2(\theta_2|y)&=&\int \pi(\theta_1,\...
Christoph Hanck's user avatar
10 votes
Accepted

Sampling from an Inverse Gamma distribution

This discrepancy arises because there are two different parameterizations of the Gamma distribution and each relate differently to the Inverse Gamma distribution. On Wikipedia, the two ...
Greenparker's user avatar
  • 15.6k
10 votes

Conditional distribution of $\exp(-|x|-|y|-a \cdot |x-y|)$

The conditional density kernels are: $$\begin{equation} \begin{aligned} f(x|y) &\propto \exp(-|x|-a \cdot |x-y|), \\[6pt] f(y|x) &\propto \exp(-|y|-a \cdot |x-y|). \\[6pt] \end{aligned} \end{...
Ben's user avatar
  • 126k
9 votes
Accepted

Stan $\hat{R}$ versus Gelman-Rubin $\hat{R}$ definition

I followed the specific link given for Gelman & Rubin (1992) and it has $$ \hat{\sigma} = \frac{n-1}{n}W+ \frac{1}{n}B $$ as in the later versions, although $\hat{\sigma}$ replaced with $\hat{\...
Aki Vehtari's user avatar
8 votes
Accepted

Gibbs sampling on the use of gamma distribution

What you are describing is a simple regression model $$ Y = X\beta + \varepsilon $$ that can be alternatively described as $$ \begin{align} \mu &= X\beta \\ Y &\sim \mathcal{N}(\mu, \tau) \...
Tim's user avatar
  • 139k
8 votes

Gibbs sampling an Ising model with 0s and 1s

The Ising model is one of the simplest examples of distributions with intractable normalising constant: the exact definition of the pmf is $$\pi(x) \propto \exp\left\{-\beta \sum_{i=1}^{19} |x_{i+1}-...
Xi'an's user avatar
  • 106k
8 votes
Accepted

Conditional distribution of $\exp(-|x|-|y|-a \cdot |x-y|)$

Disclaimer: although there is nothing to complain about Ben's answer (!), except maybe that the normalising constant of the conditional is not of direct use, here is what I wrote while being off-...
Xi'an's user avatar
  • 106k
8 votes
Accepted

Gibbs Sampler for Normal and Inverse Gamma Distribution in R

Several conceptual and R-coding errors: $\mu_0$ and $\sigma_0$ are hyper-parameters for the prior, hence should not be modified along iterations the conditionals in the Gibbs sampler are about the ...
Xi'an's user avatar
  • 106k
7 votes
Accepted

Gibbs sampling and conditional distribution

This is a question about deriving the full conditional(s) from a joint pdf rather than about Gibbs sampling. When you consider the joint distribution of the model $$ f(y,\mu,\alpha,\sigma_a^2,\...
Xi'an's user avatar
  • 106k
7 votes
Accepted

Reversibility in MCMC

I am interpreting your question as much more general in that "Is there any gain to using a reversible Markov chain over a non-reversible Markov chain?". Here are two reasons I can think of off the top ...
Greenparker's user avatar
  • 15.6k
6 votes

Metropolis-Within-Gibbs sampling with only marginal distribution known for a subset of variables

In an ideal world, sampling from $p_1(x_1)$ and then from $p_{1|2}(x_2|x_1)$ is a correct way to simulate from the joint. In case one of these distributions is unavailable, simulating a single step of ...
Xi'an's user avatar
  • 106k
6 votes
Accepted

What is the difference between monte carlo integration and gibbs sampling?

Monte Carlo integration is a technique for numerically integrating a function by evaluating it at many randomly chosen points. It's useful for computing integrals when a closed form solution doesn't ...
user20160's user avatar
  • 32.7k
6 votes

Conditional distribution for Gibbs sampling for Gaussian mixture

The Gibbs steps for a mixture model are to be found in all papers and books addressing Bayesian inference on mixtures, from our early paper with Diebolt (1990) to the reference book of Sylvia ...
Xi'an's user avatar
  • 106k
6 votes

Gibbs algorithm using negative binomial produces NAs

In short, the problem behind the question is statistically meaningless if probabilistically interesting. When using the definition of the Negative Binomial random variate as the number of failures, ...
Xi'an's user avatar
  • 106k
6 votes
Accepted

Monte Carlo Options for Data Augmentation

As noted p. 530 in the Tanner & Wong (1987) paper, the method applies to $$p_i(\theta|y) = m^{-1}\sum_{j=1}^m p(\theta|z^{(j)},y)$$ even when $m=1$. In this special case, the algorithm is a Gibbs ...
Xi'an's user avatar
  • 106k
6 votes
Accepted

How to sample using Gibbs with a uniform latent variable?

If $u_i$ has to be between $e^{-z}$ and $1$, then $e^{-z}$ needs to be between $0$ and $u_i$. Thus the distribution of $z$ given $\mathbf u$ is effectively a standard exponential RV with truncation: \...
knrumsey's user avatar
  • 8,172
5 votes
Accepted

How does Gibbs sampling produce values for a variable using the univariate conditional probability?

You raised two broad questions How expensive is Gibbs sampling? In your multivariate Gaussian example, the domains don't seem to match up, since the values obtained via Gibbs sampling seem to be ...
Greenparker's user avatar
  • 15.6k
5 votes
Accepted

Gibbs sampling and Conjugate Priors

You don't require conjugate priors for Gibbs sampling. What you need to be able to do is produce samples from the full conditional distributions. Conjugate priors generally make that easier, but ...
Glen_b's user avatar
  • 284k
5 votes
Accepted

Sampling a random binary matrix with "Gaussian" probability distribution

The target probability mass function is of the form $$\begin{align}p(A)&\propto \exp\left\{-\frac 12 \mathrm{tr}\left[\left(A-M\right)^TV\left(A-M\right)\right]\right\}\\ &\propto \exp\left\{-\...
Xi'an's user avatar
  • 106k
5 votes

The Harris recurrence of a stepping-out slice-sampling-within-Gibbs MCMC

I prove that the stepping-out and shrinkage procedure satisfies detailed balance, but this is of course not enough to show irreducibility or ergodicity. And it's easy to construct examples in which ...
Radford Neal's user avatar
5 votes
Accepted

handling metropolis hastings rejection during a Gibbs sweep

As you state, the "first part uses Metropolis Hastings to find the next parameter value" this means the exact simulation from the posterior is replaced with one (or several) Metropolis Hastings ...
Xi'an's user avatar
  • 106k
5 votes

Is Coordinate Ascent algorithm related to Gibbs Sampling in some way?

The purpose of coordinate ascent is to maximize a function, whereas the purpose of Gibbs sampling is to draw samples from a probability distribution. The two methods are loosely related in the sense ...
user20160's user avatar
  • 32.7k
5 votes
Accepted

Gibbs sampling from conditional full posterior distribution

\begin{align*} p(\beta|\sigma^{2},\gamma,\Sigma_{\gamma},y) &\propto \prod\limits_{ij} N(y_{ij}|\gamma_{i2} + z^{T}_{ij}\beta_{j},\sigma^{2}) \prod\limits_{i}N(\gamma \mid 0,\Sigma_{\gamma})\\ &...
Xi'an's user avatar
  • 106k

Only top scored, non community-wiki answers of a minimum length are eligible