Questions tagged [gibbs]

The Gibbs sampler is a simple form of Markov Chain Monte Carlo simulation, widely used in Bayesian statistics, based on sampling from full conditional distributions for each variable or group of variables. The name comes from the method being first used on Gibbs random fields modeling of images by Geman and Geman (1984).

Filter by
Sorted by
Tagged with
2
votes
1answer
38 views

Do I need to evaluate acceptance rates in Metropolis within Gibbs algorithm?

Consider the Gibbs sampler Sample $\theta' \sim p(\theta|\tau, D)$ Sample $\tau' \sim p(\tau|\theta', D)$ where $\theta,\tau$ parameters of the data $D$. Now assume that we can only sample from $p(\...
2
votes
1answer
207 views

What does it means of Normalization term of Gibbs distribution?

I am studying about Gibbs distribution concept and I am confusing a one term in that concept that is normalization term. According to the Hammersley–Clifford theorem, an random $x$ can equivalently be ...
1
vote
0answers
28 views

Sampling states of an “unnatural” Hamiltonian System

I would like to sample from a Gibbs distribution given by $$f(p, q) = \frac{1}{\mathcal{Z}}e^{-H(p, q; \omega, J)}$$ where $H$ is the Hamiltonian on generalized coordinates $(p,q)\in \mathbb{R}^{2n}...
2
votes
2answers
163 views

How does this Sampler work for the Concentration parameter of Dirichlet Process?

I am puzzled by how this Gibbs sampler on section 6 of Escobar & West (1995) works. To put it in simple words, the aim is to sample $\alpha$. The defined terms are: $$\eta\sim \texttt{Beta}(a,b)$$ ...
1
vote
0answers
21 views

Why the nodes in a Boltzmann machine need to be sampled one at a time?

Typically, we use Gibbs sampling to update (or generate samples from) energy based models. This means we update each node while keeping its markov blanket constant. Why can't we update/sample all ...
0
votes
1answer
47 views

What does MCMC do during burn-in period?

I am studying mcmc and I am wondering what mcmc does during burn-in period. And also what is the difference during burning period and after the burn-in period?
1
vote
1answer
40 views

What is the role of simulated annealing in Gibbs sampling?

While I was reading about Gibbs sampling, I happened to see "simulated annealing" but what is it doing in Gibbs sampling? Although I don't understand the full context of simulated annealing, I am ...
0
votes
0answers
25 views

Does MCMC Gibbs sampling algorithm first build a steady Markov Chain, then does the sampling to build the posterior distribution?

I am currently studying MCMC Gibbs sampling and while reading this part, a question has come into my head if MCMC Gibbs sampling first build a steady Markov Chain and does the sampling or does ...
2
votes
2answers
54 views

Need to understand a statement for Random Walk Metropolis algorithm's proposal distribution?

I was told that the proposal distribution of Random Walk Metropolis needs to be symmetric. But today I was reading a book about Bayesian Analysis which contains the following statement: "The proposal ...
1
vote
0answers
36 views

Bayesian Gamma Regression Update

I'm looking for a resource that explains how to do update the coefficients for a Bayesian gamma regression using Gibbs sampling. Specifically, if $y_i \sim Gamma(\alpha,\beta_i)$ and my data ...
36
votes
1answer
13k views

What is the difference between Metropolis-Hastings, Gibbs, Importance, and Rejection sampling?

I have been trying to learn MCMC methods and have come across Metropolis-Hastings, Gibbs, Importance, and Rejection sampling. While some of these differences are obvious, i.e., how Gibbs is a special ...
0
votes
1answer
157 views

How to create a distribution and sample?

Suppose we are given some small set of data on bundles of electrical wires and increasing voltages run through them, and we note how many of the individual wires fail. So for example, a large data ...
0
votes
0answers
26 views

Gibbs sampling for simple posterior distribution?

I have a likelihood function, $$ p(x) = \theta^{\sum x} (1- \theta)^{n-\sum x} $$ and prior distribution, $$ p(\theta) \propto \theta^{\alpha - 1} (1- \theta)^{\beta - 1}$$ then the posterior ...
-1
votes
1answer
325 views

Inferring GMM parameters with Gibbs Sampling

On my book, "Machine Learning A Probabilistic Approach". It's stated that is straightforward to derive a Gibbs sampling algorithm to fit a mixture model, especially if we use conjugate priors. So ...
0
votes
0answers
11 views

Belief Propagation vs Gibbs Sampling

In general what are the cons and pros of using Gibbs sampling to estimate a complex posterior (assuming we can sample from the conditionals) over belief propagation (using a factor graph)?
1
vote
0answers
180 views

How do I check or validate the RBM (Restricted Boltzmann Machine) Model?

I'm trying to implement RBM, then i used play tennis case to test the rbm. I've tried autoencoder before, and the result was good. Actually, I confuse with the function of RBM it self, i think it ...
1
vote
0answers
17 views

Markov chain Monte Carlo, Mixing Time

How do you estimate the mixing time for a markov chain? I read somewhere one can use the sum of the auto-correlation coefficients or the sum of the auto-covariance coefficients, but I cannot seem to ...
2
votes
0answers
221 views

Missing data in Gibbs sampling for dynamic linear models

Suppose I have the following DLM: $x_t = \Phi x_{t-1} + w_t$ $y_t = A x_t + v_t$ $x_0 \sim N(\mu_0,\Sigma_0)$ $w_t \sim N(0,Q)$ $v_t \sim N(0,R)$ Let $\Theta = \{\mu_0,\Sigma_0,\Phi,Q,A,R\}$. I ...
0
votes
0answers
18 views

sampling question in Gibbs sampling for a Gaussian mixture model

I have some confusions regarding the Gibbs sampling step for the following mixture model: consider a mixture model of the following generative process: $\theta \sim Dir(\alpha) $ (global hidden ...
1
vote
0answers
45 views

What's the point of Gibbs Sampling? [duplicate]

I am reading a book on doing Bayesian Data Analysis. I have just learned what the Metropolis Hastings (MH) Algo does, at least in relation to Bayesian Data Analysis. My understanding of the MH Algo ...
0
votes
0answers
19 views

the convergence speed for a Markov chain

For a metropolis hastings algorithm, suppose that the stationary distribution is defined as the Gibbs Boltzmann distribution $\pi_T(x)= \frac{1}{Z_T}e^{-\frac{V(x)}{T} }$ where $Z_T = \sum_{y\in V} e^{...
0
votes
1answer
39 views

Inference of the collapsed Gibbs sampling for LDA

I am trying to understand the inference procedure of collapsed Gibbs sampling for LDA model. I refer to this document and LDA wiki page. I cannot figure out how does it simplify the sample equation ...
0
votes
0answers
18 views

Dirichlet Process mixture model with independent features

I'm trying to construct a Dirichlet process mixture model for clustering where the samples have independent features. In other words, to evaluate the likelihood of sample $x_i$, I would compute $\...
0
votes
1answer
21 views

Gibb's sampling where target prob distribution is itself a conditional joint distribution - p(x,y|t)

I'm new to Gibb's sampling and need basic guidance. Say p1,p2,q-> are Gaussian variables. p1->q<-p2 and q->x where x is a discrete variable (either 1 or 0). How do I go about Sampling (using ...
1
vote
1answer
24 views

Assign an error to the parameters of MAP estimate

Through a MCMC Gibbs sampler I obtain $M$ chains of the parameters vector $\mathbf{\theta}$, meaning that each component of $\mathbf{\theta}$ is the value of one parameter at a given iteration. ...
0
votes
1answer
39 views

Gibbs sampling: Conjugate prior for a two component known-unknown variance?

If I have a normally distributed variable $\mathcal{N}(\mu,\frac{1}{\tau})$ with fixed $\mu$ then the conjugate prior for an unknown $\tau$ is then $\mathcal{Ga}(\frac{n}{2}+\alpha, \beta + \sum_i \...
0
votes
1answer
125 views

GIBBS sampling : do samples for the subset of variables approxiamate the related marginal distribution?

I'm reading the page Gibbs sampling on wikipedia. I really don't understand why the following statement is true. "The marginal distribution of any subset of variables can be approximated by simply ...
1
vote
0answers
25 views

Is there something wrong with my Bayesian hierarchical spatio-temporal model?

I built a Bayesian spatio-temporal model and one of the parameters to be estimated is the random spatial effects s. The random spatial effect is assigned an intrinsic conditional autoregressive prior (...
1
vote
1answer
583 views

Influence of word counts from DTM on LDA with Gibbs Sampling

I'm trying to wrap my head around Topic Modeling based on LDA with Gibbs sampling (Griffiths, Steyvers 2004: Finding Scientific Topics). What struck me when reading some Python implementations like ...
4
votes
1answer
87 views

Is the joint distribution $P_{XY}(x,y)$ determined from the conditionals $P_{X|Y}(x|y)$ and $P_{Y|X}(y|x)$?

For simplicity assume that $X,Y$ are discrete, finite, random variables, with joint distribution $P_{XY}(x,y) = \mathbb{P}(X=x\wedge Y=y)$. Now suppose that we do not know $P_{XY}(x,y)$, but are ...
43
votes
4answers
7k views

OpenBugs vs. JAGS

I am about to try out a BUGS style environment for estimating Bayesian models. Are there any important advantages to consider in choosing between OpenBugs or JAGS? Is one likely to replace the other ...
0
votes
0answers
36 views

sampling from joint distribution to recover marginal distribution

I'm going through Bayesian Core and have gotten stuck at this remark on page 233: " A first remark that motivates the use of the Gibbs sampler is that, within structures such as $$ \pi(x_1) = \int \...
8
votes
1answer
81 views

What is the correct way to write the elastic net?

I am confused about the correct way to write the elastic net. After reading some research papers there seems to be three forms 1) $\exp\{-\lambda_1|\beta_k|-\lambda_2\beta_k^2\}$ 2) $\exp\{-\frac{(\...
0
votes
0answers
18 views

Any relationship between MAP full conditionals and MAP joint?

Context: Bayesian model that one can draw posterior samples from via Gibbs sampling of the relevant full conditionals. Question: Can anything be said (i.e. bounds, conditions for equality) about the ...
0
votes
0answers
14 views

Gibbs Sampling attempt at a simple Coxian distribution

I have the following Coxian model for inter-arival times ($x_i$) that has $C_x^2 < 1$: $$ p(x_i\mid \lambda,\theta) = \theta \lambda^r x_i^r e^{-\lambda x_i} + (1-\theta)\lambda e^{-\lambda x_i} $...
1
vote
0answers
38 views

Understand the Holmes and Held (2006) Bayesian probit MCMC algorithm

Holmes and Held (2006) suggest a simple approach to reduce autocorrelation in the MCMC algorithm proposed by Albert and Chib (1993). HH (2006) propose to update $\beta$ and $z$ jointly, making use of ...
0
votes
0answers
65 views

Gibbs Sampler for GMM

In Rasmussen's paper it is introduced a Gibbs sampler to make inference about a standard Gaussian Mixture Model. To simplify, assume the 1-d case with basic hierarchical structure, that is: $x_i|...
3
votes
1answer
732 views

Gibbs sampling and Conjugate Priors

Are conjugate priors required when performing Gibbs sampling?
2
votes
0answers
29 views

Gibbs sampling for mixture with Dirichlet prior?

I want to sample from the distribution of a mixture distribution. The hierarchical model is $x_i\sim f$, where: $$f(x\mid \theta_1,\dots,\theta_p, w_1,\dots,\omega_p) = \sum_{j=1}w_p\varphi(x\mid\...
0
votes
0answers
28 views

Shape of parameters marginal posterior in hierarchical Bayes model

Consider a generic hierarchical Bayes model with data $y_i\sim p(y|\theta_i)$, dependent of parameters $\theta_i\sim p(\theta|\phi)$ and hyperparameters $\phi\sim p(\phi)$. Furthermore, assume that $\...
0
votes
1answer
60 views

Clusters keep switching in Gibbs sampling of Dirichlet Process Mixture Model

All the code and data for this question is on GitHub (stackexchange.R script). I've got multivariate Bernoulli data that I'd like to analyse using Bayesian Mixture ...
3
votes
1answer
56 views

Auxiliary variable Gibbs sampler

Suppose we want to sample from a pdf $f(x_1,x_2)$. It's easy to sample from $x_1 \vert x_2$, but not $x_2 \vert x_1$, so we introduce an auxiliary variable $u$ such that $\int f(x_1,x_2,u) du = f(x_1,...
1
vote
2answers
152 views

Gibbs algorithm using negative binomial produces NAs

I have the following full conditionals distributions: $$ X_2|X_1=x_1\sim Bin(x_1,p)\\ X_1|X_2=x_2\sim NegBin(x_2,p) $$ So I'm using the following code to generate a sample from each one: ...
0
votes
0answers
55 views

Metropolis-within-Gibbs for parametric inference of a regressive model

I have a regressive model of this form \begin{equation} y=f(\theta)+\varepsilon \end{equation} to describe observations $y$, with noise $\varepsilon$ and a parametric function $f$ with parameters $\...
0
votes
0answers
54 views

Can Gibbs Sampling be used to generate synthetic data from the posterior distribution?

I am working with a few datasets. I would like to expand some of them somehow by creating synthetic data. Is it possible to use a Gibbs Sampler to, by sampling a given distribution of the original ...
0
votes
1answer
29 views

Does Gibbs sampling require to know the partition function?

As the title suggest, does gibbs sampling require to know the partition function? For example, if I want to sample variable $a$ and I have worked out $p(a|rest) \propto f(a|rest)$ where $rest$ ...
0
votes
0answers
25 views

Conjugate prior for DPGMMs using Gibbs sampling

I am using Gibbs sampling to infer DPGMMs. The prior for multivariate Gaussians is Normal-inverse Wishart. But it turns out that the covariances are not estimated accurately. Here is codes and results....
0
votes
2answers
269 views

How is the accuracy of the results of Gibbs sampler measured?

How is the accuracy of the results of Gibbs sampler measured? Most resources merely say to iterate it $k$ times. But how does one infer the accuracy of the result?
2
votes
0answers
38 views

FFBS algorithm for estimating mean log-return parameter in stochastic volatility jump model

I am currently attempting to replicate this model: https://arxiv.org/pdf/1809.01501.pdf in r. My (first) problem is regarding how to sample from conditional posterior for mu, $(μ_{(j)}|Y_n, J_{(j−1)}...
3
votes
1answer
100 views

What's the role of the data in Gibbs Sampling?

Trying to wrap my mind around Gibbs Sampling. Across many answers in this same forum, I constantly notice that the examples shown do not actually require an observed data set (First example (with R ...

1 2 3 4 5 7