Questions tagged [gibbs]

The Gibbs sampler is a simple form of Markov Chain Monte Carlo simulation, widely used in Bayesian statistics, based on sampling from full conditional distributions for each variable or group of variables. The name comes from the method being first used on Gibbs random fields modeling of images by Geman and Geman (1984).

Filter by
Sorted by
Tagged with
2
votes
1answer
293 views

Conditional distribution for Gibbs sampling for Gaussian mixture

If we draw $n$ i.i.d. points $x_1,x_2,\dots,x_n$ from the following Gaussian mixture: $$ \frac 12 \mathcal N(x \mid \mu_1,1) + \frac 12 \mathcal N(x\mid \mu_2,1) $$ and the prior $p(\mu_1 , \mu_2 )$ ...
1
vote
1answer
43 views

Sampling from partial posterior

I'm reading a paper where the authors have something like the following as one step in their MCMC: $$ \begin{align} y&=\rho_1 x_1+\rho_2 x_2+\epsilon\\ z&=\beta\tilde{y}+u \end{align} $$ ...
4
votes
2answers
2k views

How to find conditional distributions from joint

I want to learn about how to do Gibbs sampling, starting with finding conditional distributions given a joint distribution. While looking for examples, I found this blog post that I wanted to ...
1
vote
0answers
44 views

HDP: Gibbs sampler implementation

I am trying to recreate the model proposed by Gao et al. (2011), based on the Hierarchical Dirichlet Process proposed by Teh and al. (2005). To estimate the model (let's call it iHDP) I need to ...
2
votes
0answers
89 views

Can Bayesian Optimization solve this problem?

Suppose ${\bf{x}} = (x_1,\ldots,x_n)$ and $f({\bf{x}})\propto 1_A({\bf{x}}) \prod_{i=1}^n {x_i}^{\alpha_i-1} e^{-\beta_i x_i}$ , i.e. $f$ is proportional to the product of independent gamma ...
1
vote
2answers
614 views

Any good book for learning probability programming

Are there any good books for me to learn probability programming? For example, I am new to Latent Dirichlet allocation (LDA) and Gibbs sampling. I have read some books about the techniques, but it ...
4
votes
1answer
38 views

Find joint maximum of a sampled density

I used a sampling method to fit a model with three parameters to data, by supplying the likelihood function and priors. (I'm using JAGS but I think this applies to any method). I obtain triplets of ...
0
votes
1answer
440 views

What is the difference between monte carlo integration and gibbs sampling?

I am aware that both are methods of sampling from the posterior. MC integration replaces the integral by a sample MC sample. Is this sample independent? Gibbs sampling is a class of MCMC ...
16
votes
1answer
8k views

When would one use Gibbs sampling instead of Metropolis-Hastings?

There are different kinds of MCMC algorithms: Metropolis-Hastings Gibbs Importance/rejection sampling (related). Why would one use Gibbs sampling instead of Metropolis-Hastings? I suspect there ...
1
vote
0answers
50 views

Approximately sampling $(X, Y)$ when sampling $X$ is easy

Suppose I am interested in sampling many pairs $(\mathbf X, Y)$ from some distribution $f(\mathbf x, y)$ where $\mathbf x \in \mathbb R^p$, $p$ large ; I am interested in both exact and approximate ...
0
votes
2answers
394 views

Gibbs Sampler Running Wild

So, I'm setting up a Gibbs Sampler using a multivariate normal model with a Jeffreys prior (working through the Hoff book on my own). There's also missing data to be imputed. I've checked my posterior ...
3
votes
1answer
29 views

Slice Sampling asks to draw from $f^{-1}]y,+\infty[$

Slice Sampling asks to draw uniformly from $f^{-1}]y,+\infty[$. Wikipedia page However, how can we be sure that a uniform defined over the set $f^{-1}]y,+\infty[$ is in fact proper? If I had to ...
2
votes
2answers
486 views

Can one use the RTS (Rauch–Tung–Striebel) smoother to simulate latent factors (as opposed to Carter and Kohn procedure)?

It is a little surprising that I have not found anything online in the literature which clarified this. I am working on an MCMC Gibbs sampling procedure to calibrate a "dynamic factor model". One of ...
0
votes
1answer
790 views

Beta distribution parameter estimation: method of moments

In a paper: Topics over time, method of moments was applied to estimate $\alpha$ and $\beta$ for a Beta distribution. My question is that how $\alpha$ or $\beta$ should be calculated if there are no ...
3
votes
1answer
82 views

Sampling posterior of empty cluster in GMM and Gibbs

Consider performing inference via a standard Gibbs sampler for a standard Gaussian Mixture Model (GMM) with $k$ components that are Gaussians $$\mathcal{N}(\mu_{k}, \sigma^{2}_{k})$$ where we assume ...
3
votes
1answer
153 views

Importance weight of conditioned particle in conditional SMC

In a generic particle filter, I understand the importance weights for each particle are calculated as $w_t^s \propto w_{t-1}^s \frac{p(y_t \mid z_t^s) p(z_t^s \mid z_{t-1}^s)}{q(z_t^s \mid z_{t-1}^s, ...
15
votes
1answer
1k views

Stan $\hat{R}$ versus Gelman-Rubin $\hat{R}$ definition

I was going through the Stan documentation which can be downloaded from here. I was particularly interested in their implementation of the Gelman-Rubin diagnostic. The original paper Gelman & ...
6
votes
1answer
11k views

Gibbs sampler examples in R [closed]

How can I implement Gibbs sampler for the posterior distribution, and estimating the marginal posterior distribution by making histogram?
0
votes
1answer
44 views

Distribution sampling when no analytic expression

The goal of gibbs sampling is to sample the joint distribution when this latter has not an analytic expression, by deriving the conditional distribution of each variable. So it is supposed that the ...
4
votes
2answers
417 views

Posterior computation for Laplace distribution

I am dealing with being Bayesian and looking for a closed form for a posterior for the scale parameter $\tau$ of a Laplace distribution, such that I can derive a full conditional in my Gibbs sampler. ...
2
votes
2answers
72 views

Generating samples from Gibbs method

I have a following homework in a subject called "Monte-Carlo Methods". I would be very thankful, if you could help me with this one, because I'm a bit stuck with this one .. The task is as follows:...
5
votes
1answer
2k views

Sampling from an Inverse Gamma distribution

I am using Gibbs sampling in the MCMC estimation of a stochastic volatility model. One of the posterior distributions is an Inverse Gamma distribution.I was struggling with the sampling procedure or ...
3
votes
1answer
80 views

Gibbs Sampling vs. Using Raw Probability in Contrastive Divergence

In Hinton's Practical Guide to Training Restricted Boltzmann Machines, Section 3, he discusses different situations in which one should take a sample from the Gibbs sampling process, and other ...
2
votes
0answers
144 views

Bayesian prior and posterior computation for a truncated normal

I have to deal with data in a Bayesian framework, ultimately devising a Gibbs sampler for inferring all my distributions parameters. Specifically, suppose I observe some univariate data distributed ...
1
vote
0answers
53 views

Non-Identifiable Multivariate Normal Posterior

So I have a theoretical question about what looks like, in my opinion, a multivariate normal distribution. The issue comes with the fact that the data is distributed with likelihood: Y |θ1, θ2 ∼ N(θ1 ...
2
votes
1answer
167 views

Why can Gibbs sampling outputs be used in Rao-Blackwellization?

I'm currently learning Chib (1995)'s method to calculate the marginal likelihood of a Bayesian model using Gibbs sampling outputs. I'm stuck in the Rao-Blackwellization step. Suppose $\mu$ and $\phi$...
11
votes
1answer
3k views

Gibbs sampling for Ising model

Homework question: Consider the 1-d Ising model. Let $x = (x_1,...x_d)$. $x_i$ is either -1 or +1 $\pi(x) \propto e^{\sum_{i=1}^{39}x_ix_{i+1}}$ Design a gibbs sampling algorithm to generate ...
3
votes
0answers
193 views

Is burn-in necessary for MCMC/Gibbs sampling if I have samples from the true distribution already?

Say I have some samples from a distribution $p$, and I want to get more samples using MCMC/Gibbs sampling. Since the existing samples are known from the equilibrium distribution $p$, if I use them as ...
2
votes
1answer
80 views

Why does $P(\theta_1\mid D, \theta_2) \propto P(D \mid \theta_1, \theta_2)P(\theta_1)$ hold?

Suppose that in a Bayesian framework we have observed data $D$, using independent prior distributions on the parameters of the model, denoted by $\theta_1, \theta_2$. Then, the joint posterior ...
1
vote
0answers
225 views

plot log likelihood function evolution in mcmc simulations

Is it possible to plot log likelihood function evolution in mcmc simulations? I have a mixture model and its parameters are estimated using the gibbs sampling method in r environment and using the ...
0
votes
1answer
111 views

prior for initial values of Kalman Filter

I'm studying Carter and Kohn's (1994) implementation of the Gibbs sampler for Bayesian analysis of state space models. In their paper, they assume the starting value, call it $\beta_0$, of the state ...
1
vote
1answer
319 views

When can the collapsed Gibbs sampler be implemented?

I understand Gibbs sampling is a means of statistics inference, and it seems that sometimes certain variables can be integrated out in the sampling process, known as collapsed Gibbs sampling. I really ...
5
votes
1answer
82 views

How to use Gibbs sampling when target function is known only up to normalising constant?

Assume we want to use Gibbs sampling to get an estimate of the parameter , and that we have the following expression for the conditional posterior of : If I am not mistaken, this means that the ...
3
votes
2answers
284 views

Gibbs sampling and Bayesian inference

I wanted to get a more in-depth understanding of sampling algorithms, and so I thought I could start with the very simple example of a binomial or bernoulli likelihood with a beta prior (since it is ...
1
vote
0answers
56 views

help interpreting plot of MCMC sample

I am estimating a model using MCMC (Gibbs Sampling). Because of the complexity of the model, I have been running two chains with many iterations. A plot of the draws for each parameter reveals a ...
11
votes
1answer
471 views

How to test if a cross-covariance matrix is non-zero?

The background of my study: In a Gibbs sampling where we sample $X$ (the variable of interests) and $Y$ from $P(X|Y)$ and $P(Y|X)$ respectively, where $X$ and $Y$ are $k$-dimensional random vectors. ...
1
vote
0answers
87 views

How to know whether a Gibbs sampler is irreducible? [duplicate]

How to know whether a Gibbs sampler is irreducible? I know that the Gibbs sampler in e.g. two variable case constructs a sequence of r.v.s $(X_1^{(i)}, X_2^{(i)})$ by sampling from the related ...
6
votes
1answer
2k views

Metropolis-Within-Gibbs sampling with only marginal distribution known for a subset of variables

Typically in Gibbs sampling we want to sample from a joint distribution $p(X_1, X_2, ..., X_N)$, but because the joint is hard to sample from directly, we instead achieve this by iteratively sampling ...
2
votes
0answers
200 views

Missing data in Gibbs sampling for dynamic linear models

Suppose I have the following DLM: $x_t = \Phi x_{t-1} + w_t$ $y_t = A x_t + v_t$ $x_0 \sim N(\mu_0,\Sigma_0)$ $w_t \sim N(0,Q)$ $v_t \sim N(0,R)$ Let $\Theta = \{\mu_0,\Sigma_0,\Phi,Q,A,R\}$. I ...
12
votes
1answer
296 views

Why does the redundant mean parameterization speed up Gibbs MCMC?

In Gelman & Hill (2007)'s book (Data Analysis Using Regression and Multilevel/Hierarchical Models), the authors claim that including redundant mean parameters can help speed up MCMC. The given ...
35
votes
1answer
12k views

What is the difference between Metropolis Hastings, Gibbs, Importance, and Rejection sampling?

I have been trying to learn MCMC methods and have come across Metropolis Hastings, Gibbs, Importance, and Rejection sampling. While some of these differences are obvious, i.e., how Gibbs is a special ...
4
votes
1answer
1k views

Explanation regarding Gibbs Sampling

I am new to MCMC and reading a intro paper regarding Gibbs sampling. However, there are two parts in the paper I cannot understand and get stuck. The first part is equation 2.3 in page 168. It says ...
1
vote
0answers
19 views

Averaging Across Gibbs Sampling Runs with Reduced Dimensions

I need help thinking through my approach to Gibbs Sampling of many parameters and I'd like to know if there is literature on this topic: I have a dataset with 3 dimensions: ...
3
votes
0answers
251 views

Gibbs sampling for parameter estimation

I am reading the paper by Willemsen et al (2015), "A multivariate Bayesian model for embryonic growth", Statistics in Medicine, 34:8, 1351–1365 where they define the posterior distribution as, \begin{...
2
votes
1answer
348 views

Why does detailed balance not provide a stopping criterion in MCMC?

Like I undestand MCMC sampling, the fulfillment of the detailed balance equation guarantees that our MC has reached its stationary distribution (given we ensure ergodicity). Detailed Balance is: $\...
1
vote
1answer
572 views

deriving posterior conditionals for gibbs sampling

I'm new to Bayesian inference and Gibbs sampling in general, and I'm struggling trying to derive the conditional posteriors for a particular data generating process I'm trying to model. The model I ...
2
votes
1answer
66 views

Generating samples for $p(\theta_{i}|\pmb{x})$ if samples from $p(\phi|\pmb{x})$ are known

Suppose $X_{i}|\theta_{i} \sim D_{1}(\theta_{i})$ and $\theta_{i}|\phi \sim D_{2}(\phi)$. Moreover $\phi \sim D_{3}(c)$ where c is known. How would I generate samples for $p(\theta_{i}|\pmb{x})$ if I ...
3
votes
0answers
277 views

Convergence of approximate Gibbs sampling

We have a bivariate random variable $(X,Y)$ for which sampling is challenging. If we were to know how to sample from the conditionals $(X|Y)$ and $(Y|X)$, we could get samples from the joint using ...
2
votes
1answer
291 views

Gibbs sampling version for estimating the Dynamic Topic Model (DTM)?

The paper of Blei et Lafferty published at ICML'06 implements a (quite complicated) variational inference (VI) technique for estimating the parameters of the Dynamic Topic Model, see: http://pdf....
0
votes
1answer
219 views

Gibbs sampler transition kernel proof [duplicate]

I'm trying to understand how the Gibbs sampling algorithm works. I've simplified it into a bivariate case to help my understanding but I'm unsure how to go from conditioning on $X^{t-1},Y^{t-1},X^t$ ...