Hot answers tagged

60 votes

What is the difference between Metropolis-Hastings, Gibbs, Importance, and Rejection sampling?

As detailed in our book with George Casella, Monte Carlo statistical methods, these methods are used to produce samples from a given distribution, with density $f$ say, either to get an idea about ...
user avatar
  • 93.5k
33 votes
Accepted

When would one use Gibbs sampling instead of Metropolis-Hastings?

Firstly, let me note [somewhat pedantically] that There are several different kinds of MCMC algorithms: Metropolis-Hastings, Gibbs, importance/rejection sampling (related). importance and ...
user avatar
  • 93.5k
27 votes

Gibbs sampler examples in R

Problem Suppose $Y \sim \text{N}(\text{mean} = \mu, \text{Var} = \frac{1}{\tau})$. Based on a sample, obtain the posterior distributions of $\mu$ and $\tau$ using the Gibbs sampler. Notation $ \...
user avatar
  • 20.2k
20 votes
Accepted

Does the Gibbs Sampling algorithm guarantee detailed balance?

You tried to show detailed balance for the Markov chain that is obtained by considering one transition of the Markov chain to be the 'Gibbs sweep' where you sample each component in turn from its ...
user avatar
  • 7,613
15 votes
Accepted

Gibbs sampling versus general MH-MCMC

the main rationale behind using the Metropolis-algorithm lies in the fact that you can use it even when the resulting posterior is unknown. For Gibbs-sampling you have to know the posterior-...
user avatar
15 votes
Accepted

Is Gibbs sampling an MCMC method?

The algorithm that is now called Gibbs sampling forms a Markov-chain and uses Monte-Carlo simulation for its inputs, so it does indeed fall within the proper scope of MCMC (Markov-Chain Monte-Carlo) ...
user avatar
  • 100k
14 votes

Can someone explain Gibbs sampling in very simple words?

I find this document GIBBS SAMPLING FOR THE UNINITIATED by Resnik & Hardisty very useful for non-statistics background folks. It explains why & how to use Gibbs sampling, and has examples ...
user avatar
  • 141
11 votes
Accepted

How to derive Gibbs sampling?

Computing a joint distribution from conditional distributions in general is very difficult. If the conditional distributions are chosen arbitrarily, a common joint distribution might not even exist. ...
user avatar
  • 5,812
11 votes
Accepted

Is the joint distribution $P_{XY}(x,y)$ determined from the conditionals $P_{X|Y}(x|y)$ and $P_{Y|X}(y|x)$?

This characterisation of the joint by the conditionals is the Hammersley-Clifford theorem. Unpublished by the authors but later established under the positivity condition by Julian Besag in 1974. The ...
user avatar
  • 93.5k
10 votes
Accepted

How to find conditional distributions from joint

Those distributions you call "marginal" are not marginal. They are conditional distributions because you wrote $x \mid y$. The marginal distribution of $X$, for example, is necessarily independent ...
user avatar
  • 5,056
10 votes

Why does sampling from the posterior predictive distribution $p(x_{new} \mid x_1, \ldots x_n)$ work without having to average out the integral?

What you are actually doing with the two-step process you've outlined is sampling from the joint distribution $p(x_{new}, \mu \thinspace | \thinspace x_1, \dots, x_n)$, then ignoring the sampled ...
user avatar
  • 33.7k
10 votes
Accepted

Rao-Blackwellization of Gibbs Sampler

Assuming I take the mean of the posterior distribution rather than a random sample from it, is this what is commonly referred to as Rao-Blackwellization? I am not very familiar with stochastic ...
user avatar
  • 14.4k
10 votes
Accepted

Sampling from an Inverse Gamma distribution

This discrepancy arises because there are two different parameterizations of the Gamma distribution and each relate differently to the Inverse Gamma distribution. On Wikipedia, the two ...
user avatar
  • 14.4k
10 votes
Accepted

Posterior computation for Laplace distribution

The likelihood for $n$ iid observations looks like: $ f(x_1,...x_n|\lambda,\mu) \propto \frac{1}{\lambda^n} exp(-\frac{1}{\lambda}\sum_{i=1}^n|x_i-\mu|)$ Hence a conjugate prior for $\lambda$ with $\...
user avatar
  • 4,031
10 votes

Conditional distribution of $\exp(-|x|-|y|-a \cdot |x-y|)$

The conditional density kernels are: $$\begin{equation} \begin{aligned} f(x|y) &\propto \exp(-|x|-a \cdot |x-y|), \\[6pt] f(y|x) &\propto \exp(-|y|-a \cdot |x-y|). \\[6pt] \end{aligned} \end{...
user avatar
  • 100k
9 votes

Rao-Blackwellization of Gibbs Sampler

The Gibbs sampler can then be used to improve efficiency of (say) samples from a marginal posterior, call it $\pi_2(\theta_2|y)$. Note \begin{eqnarray*} \pi_2(\theta_2|y)&=&\int \pi(\theta_1,\...
user avatar
8 votes
Accepted

Confusion in Gibbs sampling

Since I'm not sure where are you stuck at, I'll try multiple shots: Explanation 1: The thing is that you only need the form of the unnormalized posterior, and that is why it's enough if you can get: ...
user avatar
  • 2,696
8 votes
Accepted

MCMC chain getting stuck

In step 4, you don't have to reject the proposal $x,\theta$ every time its new likelihood is lower; if you do so, you are doing a sort of optimization instead of sampling from the posterior ...
user avatar
  • 2,696
8 votes
Accepted

Bayesian regression full conditional distribution

As you show in the reproduction what is written in this book, the solution is incorrect for the simple reason that the quantity $(X^{T}X)^{-1}$ is a $p\times p$ matrix, not a scalar. Hence you cannot ...
user avatar
  • 93.5k
8 votes
Accepted

Gibbs sampling from a complex full conditional

Here is an excerpt from our Monte Carlo Statistical Methods book: 10.3.3. Metropolizing the Gibbs Sampler Hybrid MCMC algorithms are often useful at an elementary level of the simulation process; ...
user avatar
  • 93.5k
8 votes

Gibbs sampling an Ising model with 0s and 1s

The Ising model is one of the simplest examples of distributions with intractable normalising constant: the exact definition of the pmf is $$\pi(x) \propto \exp\left\{-\beta \sum_{i=1}^{19} |x_{i+1}-...
user avatar
  • 93.5k
8 votes
Accepted

Conditional distribution of $\exp(-|x|-|y|-a \cdot |x-y|)$

Disclaimer: although there is nothing to complain about Ben's answer (!), except maybe that the normalising constant of the conditional is not of direct use, here is what I wrote while being off-...
user avatar
  • 93.5k
7 votes
Accepted

Marginal Likelihood from the Gibbs Output

There is a slight programming mistake in the prior ...
user avatar
  • 93.5k
7 votes

Metropolis-Hastings Algorithm within Gibbs Sampling

Are you sure the joint density$$f(x_1,x_2)=\left(\dfrac{x_1}{x_2}\right)\left(\dfrac{\alpha}{x_2}\right)^{x_1-1}\exp\left\{-\left(\dfrac{\alpha}{x_2}\right)^{x_1} > \right\}\mathbb{I}_{\mathbb{R}^*...
user avatar
  • 93.5k
7 votes
Accepted

Why does Slice sampler use the log of the density?

The slice sampler does not "sample from the log-density". It can, however, use the log density in the calculations to obtain a dependent sequence of observations from the density. The basic idea of a ...
user avatar
  • 262k
7 votes
Accepted

Gibbs sampling and conditional distribution

This is a question about deriving the full conditional(s) from a joint pdf rather than about Gibbs sampling. When you consider the joint distribution of the model $$ f(y,\mu,\alpha,\sigma_a^2,\...
user avatar
  • 93.5k
7 votes
Accepted

Gibbs sampling on the use of gamma distribution

What you are describing is a simple regression model $$ Y = X\beta + \varepsilon $$ that can be alternatively described as $$ \begin{align} \mu &= X\beta \\ Y &\sim \mathcal{N}(\mu, \tau) \...
user avatar
  • 117k
7 votes
Accepted

Stan $\hat{R}$ versus Gelman-Rubin $\hat{R}$ definition

I followed the specific link given for Gelman & Rubin (1992) and it has $$ \hat{\sigma} = \frac{n-1}{n}W+ \frac{1}{n}B $$ as in the later versions, although $\hat{\sigma}$ replaced with $\hat{\...
user avatar
7 votes
Accepted

Gibbs Sampler for Normal and Inverse Gamma Distribution in R

Several conceptual and R-coding errors: $\mu_0$ and $\sigma_0$ are hyper-parameters for the prior, hence should not be modified along iterations the conditionals in the Gibbs sampler are about the ...
user avatar
  • 93.5k
6 votes
Accepted

Zero-inflated Poisson and Gibbs sampling, proofs and sampling

For the first question, given that you have a hierarchical model, you simply have to start from the higher level and proceed downwards: $(r_i|p, \lambda )\sim\mathcal{B}(p)$ for $i=1,\ldots,n$ $(x_i|...
user avatar
  • 93.5k

Only top scored, non community-wiki answers of a minimum length are eligible