Stack Exchange Network

Stack Exchange network consists of 174 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange
Join us in building a kind, collaborative learning community via our updated Code of Conduct.

The Gibbs sampler is a simple form of Markov Chain Monte Carlo simulation, widely used in Bayesian statistics, based on sampling from full conditional distributions for each variable or group of variables. The name comes from the method being first used on Gibbs random fields modeling of images by ...

1
vote
0answers
10 views

Slice Sampling asks to draw from $f^{-1}]y,+\infty[$

Slice Sampling asks to draw uniformly from $f^{-1}]y,+\infty[$. Wikipedia page However, how can we be sure that a uniform defined over the set $f^{-1}]y,+\infty[$ is in fact proper? If I had to ...
3
votes
1answer
110 views

Importance weight of conditioned particle in conditional SMC

In a generic particle filter, I understand the importance weights for each particle are calculated as $w_t^s \propto w_{t-1}^s \frac{p(y_t \mid z_t^s) p(z_t^s \mid z_{t-1}^s)}{q(z_t^s \mid z_{t-1}^s, ...
0
votes
0answers
19 views

Markov Chain Monte Carlo [duplicate]

what are the differences between M-H algorithm and M-H-within-Gibbs algorithm. If possible, upload for me the two algorithms please.
13
votes
1answer
522 views

Stan $\hat{R}$ versus Gelman-Rubin $\hat{R}$ definition

I was going through the Stan documentation which can be downloaded from here. I was particularly interested in their implementation of the Gelman-Rubin diagnostic. The original paper Gelman & ...
3
votes
1answer
30 views

Sampling posterior of empty cluster in GMM and Gibbs

Consider performing inference via a standard Gibbs sampler for a standard Gaussian Mixture Model (GMM) with $k$ components that are Gaussians $$\mathcal{N}(\mu_{k}, \sigma^{2}_{k})$$ where we assume ...
0
votes
0answers
15 views

Selecting the hyperparameters of the conditional distributions in gibbs sampler

Suppose you have a gibbs sampler where each $x_j|x_{-j}$ is a known truncated distribution (for example truncated beta distribution). I'm interested in selecting the hyperparameters given the true ...
0
votes
1answer
21 views

Distribution sampling when no analytic expression

The goal of gibbs sampling is to sample the joint distribution when this latter has not an analytic expression, by deriving the conditional distribution of each variable. So it is supposed that the ...
4
votes
2answers
102 views

Posterior computation for Laplace distribution

I am dealing with being Bayesian and looking for a closed form for a posterior for the scale parameter $\tau$ of a Laplace distribution, such that I can derive a full conditional in my Gibbs sampler. ...
2
votes
2answers
57 views

Generating samples from Gibbs method

I have a following homework in a subject called "Monte-Carlo Methods". I would be very thankful, if you could help me with this one, because I'm a bit stuck with this one .. The task is as follows:...
3
votes
1answer
39 views

Gibbs Sampling vs. Using Raw Probability in Contrastive Divergence

In Hinton's Practical Guide to Training Restricted Boltzmann Machines, Section 3, he discusses different situations in which one should take a sample from the Gibbs sampling process, and other ...
2
votes
0answers
34 views

Bayesian prior and posterior computation for a truncated normal

I have to deal with data in a Bayesian framework, ultimately devising a Gibbs sampler for inferring all my distributions parameters. Specifically, suppose I observe some univariate data distributed ...
1
vote
0answers
15 views

Non-Identifiable Multivariate Normal Posterior

So I have a theoretical question about what looks like, in my opinion, a multivariate normal distribution. The issue comes with the fact that the data is distributed with likelihood: Y |θ1, θ2 ∼ N(θ1 ...
2
votes
1answer
92 views

Why can Gibbs sampling outputs be used in Rao-Blackwellization?

I'm currently learning Chib (1995)'s method to calculate the marginal likelihood of a Bayesian model using Gibbs sampling outputs. I'm stuck in the Rao-Blackwellization step. Suppose $\mu$ and $\phi$...
0
votes
0answers
21 views

A conditioning simplification in a graphical clustering model

The graphical model associated with this problem is (reproduction from Murphy's Machine Learning a Probabilistic Perspective) and the author argues on page 843 of MLAPP, Intuitively, it is right. ...
3
votes
0answers
70 views

Is burn-in necessary for MCMC/Gibbs sampling if I have samples from the true distribution already?

Say I have some samples from a distribution $p$, and I want to get more samples using MCMC/Gibbs sampling. Since the existing samples are known from the equilibrium distribution $p$, if I use them as ...
2
votes
1answer
57 views

Why does $P(\theta_1\mid D, \theta_2) \propto P(D \mid \theta_1, \theta_2)P(\theta_1)$ hold?

Suppose that in a Bayesian framework we have observed data $D$, using independent prior distributions on the parameters of the model, denoted by $\theta_1, \theta_2$. Then, the joint posterior ...
0
votes
1answer
34 views

prior for initial values of Kalman Filter

I'm studying Carter and Kohn's (1994) implementation of the Gibbs sampler for Bayesian analysis of state space models. In their paper, they assume the starting value, call it $\beta_0$, of the state ...
5
votes
1answer
54 views

How to use Gibbs sampling when target function is known only up to normalising constant?

Assume we want to use Gibbs sampling to get an estimate of the parameter , and that we have the following expression for the conditional posterior of : If I am not mistaken, this means that the ...
1
vote
0answers
42 views

help interpreting plot of MCMC sample

I am estimating a model using MCMC (Gibbs Sampling). Because of the complexity of the model, I have been running two chains with many iterations. A plot of the draws for each parameter reveals a ...
0
votes
0answers
16 views

What is the meaning of the clique-function when factorising undirected graphs?

I am currently reading Graphical Models and Message-Passing Algorithms: Some Introductory Lectures and am quite at the beginning. I understand the factorization for directed graphical methods, which ...
3
votes
2answers
129 views

Gibbs sampling and Bayesian inference

I wanted to get a more in-depth understanding of sampling algorithms, and so I thought I could start with the very simple example of a binomial or bernoulli likelihood with a beta prior (since it is ...
1
vote
0answers
42 views

Burning and Sampling in Gibbs sampler [closed]

I am confused about the concept of sampling in the Gibbs sampler after the burn-in loops. This is the basic problem, I have a picture composed of 1's and 0's. This is a noisy version and I am trying ...
1
vote
1answer
92 views

When can the collapsed Gibbs sampler be implemented?

I understand Gibbs sampling is a means of statistics inference, and it seems that sometimes certain variables can be integrated out in the sampling process, known as collapsed Gibbs sampling. I really ...
1
vote
0answers
29 views

How to know whether a Gibbs sampler is irreducible? [duplicate]

How to know whether a Gibbs sampler is irreducible? I know that the Gibbs sampler in e.g. two variable case constructs a sequence of r.v.s $(X_1^{(i)}, X_2^{(i)})$ by sampling from the related ...
0
votes
2answers
56 views

How is the accuracy of the results of Gibbs sampler measured?

How is the accuracy of the results of Gibbs sampler measured? Most resources merely say to iterate it $k$ times. But how does one infer the accuracy of the result?
3
votes
1answer
335 views

Metropolis-Within-Gibbs sampling with only marginal distribution known for a subset of variables

Typically in Gibbs sampling we want to sample from a joint distribution $p(X_1, X_2, ..., X_N)$, but because the joint is hard to sample from directly, we instead achieve this by iteratively sampling ...
0
votes
0answers
44 views

Missing data in Gibbs sampling for dynamic linear models

Suppose I have the following DLM: $x_t = \Phi x_{t-1} + w_t$ $y_t = A x_t + v_t$ $x_0 \sim N(\mu_0,\Sigma_0)$ $w_t \sim N(0,Q)$ $v_t \sim N(0,R)$ Let $\Theta = \{\mu_0,\Sigma_0,\Phi,Q,A,R\}$. I ...
0
votes
0answers
56 views

Gibbs sampling - Distribution of population mean and precision

I have a question when trying to implementing a simple Gibbs sampling. Given that $Y=(y_1, y_2,..., y_n)$ $Y \sim \text{N}(\text{mean} = \mu, \text{Var} = \frac{1}{\tau})$ $\mu \sim N(a,b)$ $\tau ...
4
votes
1answer
265 views

Explanation regarding Gibbs Sampling

I am new to MCMC and reading a intro paper regarding Gibbs sampling. However, there are two parts in the paper I cannot understand and get stuck. The first part is equation 2.3 in page 168. It says ...
1
vote
0answers
17 views

Averaging Across Gibbs Sampling Runs with Reduced Dimensions

I need help thinking through my approach to Gibbs Sampling of many parameters and I'd like to know if there is literature on this topic: I have a dataset with 3 dimensions: ...
2
votes
1answer
220 views

Why does detailed balance not provide a stopping criterion in MCMC?

Like I undestand MCMC sampling, the fulfillment of the detailed balance equation guarantees that our MC has reached its stationary distribution (given we ensure ergodicity). Detailed Balance is: $\...
1
vote
1answer
182 views

deriving posterior conditionals for gibbs sampling

I'm new to Bayesian inference and Gibbs sampling in general, and I'm struggling trying to derive the conditional posteriors for a particular data generating process I'm trying to model. The model I ...
2
votes
0answers
163 views

Convergence of approximate Gibbs sampling

We have a bivariate random variable $(X,Y)$ for which sampling is challenging. If we were to know how to sample from the conditionals $(X|Y)$ and $(Y|X)$, we could get samples from the joint using ...
2
votes
1answer
61 views

Generating samples for $p(\theta_{i}|\pmb{x})$ if samples from $p(\phi|\pmb{x})$ are known

Suppose $X_{i}|\theta_{i} \sim D_{1}(\theta_{i})$ and $\theta_{i}|\phi \sim D_{2}(\phi)$. Moreover $\phi \sim D_{3}(c)$ where c is known. How would I generate samples for $p(\theta_{i}|\pmb{x})$ if I ...
0
votes
1answer
91 views

Gibbs sampler transition kernel proof [duplicate]

I'm trying to understand how the Gibbs sampling algorithm works. I've simplified it into a bivariate case to help my understanding but I'm unsure how to go from conditioning on $X^{t-1},Y^{t-1},X^t$ ...
0
votes
1answer
88 views

Sampling from conditional posterior - continuous and discrete terms

I'm working through Hierarchically Supervised LDA by Perotte et all (2011). The conditional posterior I'm supposed to sample values $z_i$ from, however, is zero almost everywhere. To see why, lets ...
0
votes
0answers
35 views

Inability to Replicate Posterior for σ² of Normal Regression Model by Gibbs Sampling

For large sample sizes (N) The Gibbs Sampler (GS) does not seem to replicate the true posterior for σ² when the number of regressors (k) is greater than 2. We are trying to test Gibbs Sampling in a ...
1
vote
1answer
104 views

Sampling from a posterior with Gibbs sampling

In an image processing class, I dont really get behind the idea how to 'sample from a posterior' with Gibbs sampling. We have a posterior distribution: $f(z_1, .. ,z_n \mid x_1,.. ,x_n) := f(z \mid x)...
3
votes
0answers
172 views

Gibbs sampling for parameter estimation

I am reading the paper by Willemsen et al (2015), "A multivariate Bayesian model for embryonic growth", Statistics in Medicine, 34:8, 1351–1365 where they define the posterior distribution as, \begin{...
2
votes
1answer
341 views

Gibbs sampling an Ising model with 0s and 1s

One of my problems in one of my courses ask to sample a 20 dimensional vector of 0s and 1s, $\{0,1\}^{20},$ when they are distributed as $$ \pi(x) = \exp\left\{-\beta \sum_{i=1}^{19} |x_{i+1}-x_i| \...
1
vote
0answers
23 views

Gibbs sampling with expectations instead of sampling

I see there is something called Iterated Conditional Models (ICM), which is a sort of Gibbs sampling where, instead of sampling, we use the value that maximizes the conditional. That is: ...
0
votes
1answer
38 views

Priors on Taylor Expansion series

I'm wondering what priors can i choose for a Taylor series as follows: $\theta_{1}+\theta_{2} (y-\alpha) + \theta_{3} (y-\alpha)^2$ What priors should I use for updating these parameters ($\theta_{1},...
1
vote
0answers
180 views

Collapsed Gibbs sampler on Hierarchical Dirichlet Process Mixture Model

I am trying to design a collapsed Gibbs sampler on a mixture model based on Hierarchical Dirichlet Process ($g\sim DP(\gamma, b)$, $\pi\sim DP(\alpha, g)$ ). Should I resample from the posterior of ...
0
votes
1answer
71 views

Rewrite conditional formula with three variables using Bayes formula

In equation (5) on page 3 on this paper a conditional probability is rewritten using Bayes' formula. I started using this answer Can I rewrite conditional probability of three variables like this? ...
0
votes
1answer
387 views

Beta distribution parameter estimation: method of moments

In a paper: Topics over time, method of moments was applied to estimate $\alpha$ and $\beta$ for a Beta distribution. My question is that how $\alpha$ or $\beta$ should be calculated if there are no ...
4
votes
1answer
49 views

Sampling from multivariate normal conditional on a negative minimum

Let $X\sim \mathcal{N}(\mu,\Sigma)$, where $\mu\in\mathbb{R}^n$ and $\Sigma\in\mathbb{R}^{n\times n}$. How can I efficiently sample from $X | {\min{X}\le 0}$? (I.e. from the distribution of $X$ ...
0
votes
0answers
20 views

Gibbs sampler from from $p(x) = C g(x)$ with $C$ unknown and discrete elements in $X$

By using the Hasting-Metropolis method, is there a way to draw samples from a distribution of this form: $$p(\textbf{x}) = C g(\textbf{x})$$ For $x$ being two dimensional and discrete. The reason that ...
12
votes
1answer
222 views

Why does the redundant mean parameterization speed up Gibbs MCMC?

In Gelman & Hill (2007)'s book (Data Analysis Using Regression and Multilevel/Hierarchical Models), the authors claim that including redundant mean parameters can help speed up MCMC. The given ...
1
vote
1answer
209 views

Influence of word counts from DTM on LDA with Gibbs Sampling

I'm trying to wrap my head around Topic Modeling based on LDA with Gibbs sampling (Griffiths, Steyvers 2004: Finding Scientific Topics). What struck me when reading some Python implementations like ...
0
votes
0answers
56 views

full conditional distribution to recognize

Can some one recognize this distribution that cames out like the full conditional w.r. to gamma of a non-linear regression model? $\pi(\gamma|\alpha,\beta,\nu,y_i,x_i) \propto 1_{(0,1)}(\gamma) \...