Questions tagged [gibbs]

The Gibbs sampler is a simple form of Markov Chain Monte Carlo simulation, widely used in Bayesian statistics, based on sampling from full conditional distributions for each variable or group of variables. The name comes from the method being first used on Gibbs random fields modeling of images by Geman and Geman (1984).

Filter by
Sorted by
Tagged with
5
votes
1answer
1k views

Convergence results for block-gibbs sampling?

Suppose you have some complex model you want to sample from by Markov chain Monte Carlo. There are many types of situations where you can divide your variables into, say, two groups, and efficiently ...
5
votes
1answer
682 views

Confusion in Gibbs sampling

I am self-studying Gibbs sampling from a book. The book introduces metropolis hastings algortihm to generate representative values from a posterior distribution. So we know $p(D | \theta) p(\theta)$ ...
5
votes
2answers
283 views

Gibbs Sampling and Probability Notation

Problem 1 I am trying to implement Gibbs Sampling for the following problem: There is a grid measuring 3 x 3 sites, each "site" can be designated in a state, $X$, of 1 or -1. The sites are numbered ...
5
votes
2answers
854 views

slice sampling within a Gibbs sampler

Questions My questions are: Is the following slice-sampling-within-Gibbs approach valid? Is there a good reference out there that uses, or better yet, justifies it? Context I'm trying to sample ...
5
votes
2answers
3k views

Clustering methods for unknown number of clusters

Matrix $X=[x_1,...,x_i,...,x_N]$ is a data-set containing $N$ data-points that each data-point $x_i$ is a vector of $D$ dimensions. Each dimension is a feature. The number of clusters ($K$) is unknown....
5
votes
1answer
83 views

How to use Gibbs sampling when target function is known only up to normalising constant?

Assume we want to use Gibbs sampling to get an estimate of the parameter , and that we have the following expression for the conditional posterior of : If I am not mistaken, this means that the ...
5
votes
1answer
2k views

What are the main differences between classical and Gibbs sampling Latent Dirichlet Allocations?

In these weeks I have been studying the classical Latent Dirichlet Allocation (LDA) algorithm by David Blei and colleagues (2003), and the LDA variant based on Gibbs sampling introduced by Tom ...
5
votes
1answer
97 views

Estimation of Bayesian Models

I'm trying to get into Bayesian model estimation (I'm interested in posterior parameter distributions). I could get away with Metropolis-Hastings and Gibbs Sampling for models with few parameters (<...
5
votes
1answer
808 views

In Bayesian analysis, how to sample from full conditional given uniform prior and normal data likelihood?

[EDIT] This question comes from the example of OpenBUGS manual: Stagnant: a changepoint problem and an illustration of how NOT to do MCMC! I also asked another question regarding this example. [...
5
votes
0answers
626 views

Gibbs sampling deriving complete conditionals with mixture priors

My question is about the derivation of the complete conditionals for Gibbs sampling in a hierarchical model where some of the parameters are mixtures of point-masses and Normal distributions. The ...
5
votes
0answers
254 views

Bayesian estimates for Deming regression coinciding with least-squares estimates

Consider the following Deming model with independent replicates : $$x_{i,j} \mid \theta_{i} \sim {\cal N}(\theta_{i}, \gamma_X^2), \quad y_{i,j} \mid \theta_{i} \sim {\cal N}(\alpha+\beta\theta_{i}, \...
4
votes
2answers
509 views

Posterior computation for Laplace distribution

I am dealing with being Bayesian and looking for a closed form for a posterior for the scale parameter $\tau$ of a Laplace distribution, such that I can derive a full conditional in my Gibbs sampler. ...
4
votes
1answer
77 views

Is the joint distribution $P_{XY}(x,y)$ determined from the conditionals $P_{X|Y}(x|y)$ and $P_{Y|X}(y|x)$?

For simplicity assume that $X,Y$ are discrete, finite, random variables, with joint distribution $P_{XY}(x,y) = \mathbb{P}(X=x\wedge Y=y)$. Now suppose that we do not know $P_{XY}(x,y)$, but are ...
4
votes
2answers
304 views

Canonical example to understand Gibbs Sampling

I'm been trying to understand Gibbs sampling. What I'm looking for is a paper or other reference which uses a simple canonical example and uses that to illustrate Gibbs sampling. Sadly I've not ...
4
votes
1answer
381 views

Gibbs sampling and conditional distribution

I need to simulate the posterior distribution of intraclass correlation coefficient $\pi(\rho|y)$ where $y$ is the data set and $\rho=\frac{\sigma_a^2}{\sigma_a^2+\sigma_e^2}$ with $\sigma^2_a\sim IG(\...
4
votes
2answers
279 views

The Harris recurrence of a stepping-out slice-sampling-within-Gibbs MCMC

I want to use a multistage version of the MCMC here. That is, I want to use a Gibbs sampler to draw from a general joint distribution $p(x_1, x_2, x_3, \ldots)$ with a Gibbs step for each full ...
4
votes
2answers
2k views

A robust R package to do MCMC and Gibbs sampling

I need to make linear model for which I need to do Gibbs sampling in MCMC simulations. The model needed to be fitted is a linear mixed model. Please suggest me for a robust R package for this task.
4
votes
1answer
201 views

handling metropolis hastings rejection during a Gibbs sweep

Suppose I have a MCMC involving a 2 step Gibbs sampler. The first part uses metropolis hastings to find the next parameter value. If during one sweep, the result for the first part is a rejections, ...
4
votes
2answers
919 views

Is Slice Sampling a special case of Gibbs Sampling?

I read on this thread the following: If you can use both the gibbs sampler and slice sampling to sample from a posterior I would use the Gibbs sampler as the slice sampler seems unnecessary to ...
4
votes
2answers
363 views

Resources about Gibbs sampling in hybrid Bayesian networks

I've been trying to get my hands on a substantial resource for using Gibbs sampling in hybrid Bayesian networks, that is, networks with both continuous and discrete variables. So far I can't say I ...
4
votes
1answer
2k views

Explanation regarding Gibbs Sampling

I am new to MCMC and reading a intro paper regarding Gibbs sampling. However, there are two parts in the paper I cannot understand and get stuck. The first part is equation 2.3 in page 168. It says ...
4
votes
1answer
497 views

Reversibility in MCMC

I am reading Geyer's lecture notes on MCMC. A condensed version of these notes constitutes Chapter 1 of the Handbook of Markov Chain Monte Carlo (ed. Brooks et al., 2011). Geyer notes that the ...
4
votes
1answer
38 views

Find joint maximum of a sampled density

I used a sampling method to fit a model with three parameters to data, by supplying the likelihood function and priors. (I'm using JAGS but I think this applies to any method). I obtain triplets of ...
4
votes
1answer
182 views

Gibbs sampling, what to use?

My question concerns Gibbs sampling. Suppose that I have three unknown quantities, $\mu, \sigma^2$ and $c$. I have given prior information and I have given the likelihood which allows me to compute ...
4
votes
1answer
121 views

Gibbs sampler for a particular distribution

I'm trying to implement Gibbs Sampler for the distribution: $$\pi(x,y)=e^{-10(x^2-y)^2-(y-1/4)^4}$$ So, like the first step, I need to find: $$\phi(t) = \int_{-\infty}^{t} e^{-10(x^2-y)^2-(y-0.25)^4} ...
4
votes
1answer
915 views

JAGS, cannot evaluate upper index of counter

I asked this question at the JAGS sourceforge help forum but didn't get response there. I have the following JAGS model: ...
4
votes
1answer
388 views

What is the state-of-the-art regarding sampling from discrete distribution?

After struggling with auto-Poisson model (a.k.a. Random Markov Network with conditional Poisson distributions) trying to force Gibbs sampler to obtain discrete sample of the network (since I know ...
4
votes
2answers
790 views

Is sequential Bayesian updating an option when using MCMC?

I have an implementation of the Griddy Gibbs sampler, but my observations on which I'm conditioning model parameters are too many in number, thus the likelihood underflows quickly, even with a log ...
4
votes
1answer
91 views

Confusion related to derivation of conditional distribution

I was reading this paper http://stat.duke.edu/phd-program/gcc/resources/ResourcesDocuments/CodeExplanation.pdf related to Gibbs sampling. Suppose we have n iid samples $x_i$ from $N(\mu,\sigma^2)$ ...
4
votes
1answer
80 views

Sampling from multivariate normal conditional on a negative minimum

Let $X\sim \mathcal{N}(\mu,\Sigma)$, where $\mu\in\mathbb{R}^n$ and $\Sigma\in\mathbb{R}^{n\times n}$. How can I efficiently sample from $X | {\min{X}\le 0}$? (I.e. from the distribution of $X$ ...
4
votes
0answers
654 views

Gibbs sampling for spike and slab priors

In Spike and slab variable selection (equation 4) there is a model setup of the form $\beta_k | \lambda_k, \tau_k \sim \text{Normal} (0, \lambda_k \tau_k^2)$ $\lambda_k | \nu_0, w \sim (1-w)\delta_{\...
4
votes
0answers
505 views

Gibbs sampling for correlated random variables

Short summary Suppose two latent variables of a hierarchical model are correlated. Let $1-\epsilon$ be the degree of correlation. As $\epsilon\rightarrow 0$ the variables become perfectly correlated ...
4
votes
0answers
1k views

Gibbs sampling for LDA — does a small Dirichlet concentration parameter make a difference?

I'm using a Gibbs sampler for Latent Dirichlet allocation as described by Griffiths and Steyvers (http://www.ncbi.nlm.nih.gov/pmc/articles/PMC387300/). The sampling of a new topic $j$ for word $i$ is ...
4
votes
0answers
301 views

Convergence theorem for Gibbs sampling

The convergence theorem for Gibbs sampling states: Given a random vector $X$ with components $X_1,X_2,...X_K$ and the knowledge about the conditional distribution of $X_k$ we can find the actual ...
4
votes
0answers
4k views

Metropolis-Hastings within Gibbs sampling

Suppose we have the following classical normal linear regression model: $$y_i = \beta_1 x_{1i} + \beta_2x_{2i} + \beta_3x_{3i} + e_i$$ where $e_{i} \sim iid.N(0, \sigma^2)$ for all $i = 1, 2, \cdots,...
4
votes
0answers
298 views

Gibbs sampling from full conditionals

I have the following joint density: $p(x_1,x_2,y_1,y_2) \propto \exp\left(−\left(x_1^2+x_2^2+c_1(y_2-y_1)^2+c_2(y_2-y_1)^4\right)\right)$ Can I use Gibbs sampling to sample from that? How can I get ...
3
votes
2answers
2k views

Sampling variables and calculating likelihood in WinBUGS/OpenBUGS

I am trying to read some WinBUGS/OpenBUGS examples to figure out how to specify models. I can't seem to understand where the probabilistic dnorm, ...
3
votes
2answers
299 views

Gibbs sampling and Bayesian inference

I wanted to get a more in-depth understanding of sampling algorithms, and so I thought I could start with the very simple example of a binomial or bernoulli likelihood with a beta prior (since it is ...
3
votes
1answer
2k views

Gibbs sampling convergence

In an astronomical context, the authors of a paper desire to use a Gibbs algorithm. Please note: I am inexperience in MCMC algorithms, and specifically in Gibbs sampling. What we want, in essence, is ...
3
votes
1answer
608 views

Gibbs sampling an Ising model with 0s and 1s

One of my problems in one of my courses ask to sample a 20 dimensional vector of 0s and 1s, $\{0,1\}^{20},$ when they are distributed as $$ \pi(x) = \exp\left\{-\beta \sum_{i=1}^{19} |x_{i+1}-x_i| \...
3
votes
1answer
665 views

Gibbs sampling and Conjugate Priors

Are conjugate priors required when performing Gibbs sampling?
3
votes
1answer
267 views

Sampling a random binary matrix with “Gaussian” probability distribution

Let $A_{ij}$ be a $n\times n$ random binary matrix with probability mass function $P(A)$ given by $$ \log P(A)=-\frac 12 \mathrm{tr}\left[\left(A-M\right)^TV\left(A-M\right)\right] + C, $$ where $M$ ...
3
votes
1answer
185 views

What is a hierarchical model that can estimated via the Metropolis-Hastings Algorithm but not the Gibbs Sampler?

My understanding of the differences between MH and Gibbs Samplers is that a Gibbs Sampler is usually used when the full conditionals are present to us. In other words, it is a known distribution, so ...
3
votes
2answers
630 views

Markov Chain Monte Carlo (MCMC) with transformed data

I want to obtain an estimate of a parameter $\Theta$ in a model for a random variable $X$ dependent on $\Theta$ with known but complicated likelihood $L(\Theta|X) = p(X|\Theta)$. $X$ is not directly ...
3
votes
1answer
1k views

Gibbs sampling with mixed prior using a Metropolis-Hastings step

My questions are about a sampling procedure for fitting a Bayesian hierarchical model where one of the priors is a mixture distribution of discrete and continuous parts. The model is not my own but I ...
3
votes
1answer
511 views

Gibbs sampling with Dirichlet Likelihood

I have a sequence of observations that I am representing as proportions: X1 X2 X3 X4 X5 0.10 0.20 0.50 0.12 0.08 0.07 0.24 0.55 0.04 0.10 ... ...
3
votes
1answer
4k views

JAGS: posterior predictive distribution

I am fitting a simple linear regression model with R & JAGS: ...
3
votes
1answer
30 views

Slice Sampling asks to draw from $f^{-1}]y,+\infty[$

Slice Sampling asks to draw uniformly from $f^{-1}]y,+\infty[$. Wikipedia page However, how can we be sure that a uniform defined over the set $f^{-1}]y,+\infty[$ is in fact proper? If I had to ...
3
votes
1answer
1k views

Gibbs measure and normal distribution

On Wikipedia, the Gibbs measure defines the probability as: $$ P(X=x) = \frac{1}{Z(\beta)}\exp(-\beta E(x)) $$ Now, the familiar form of the normal distribution is: $$ P(x) = \frac{1}{\sqrt{2\pi}\...
3
votes
1answer
803 views

Prior selection for Gaussian Processes (GP)

I am trying to select a prior for the covariance parameters of my Gaussian Process (GP) and have been running into numerical problems with my MCMC code. My model is the following: $$Y = D\beta + GP(...