Questions tagged [gibbs]

The Gibbs sampler is a simple form of Markov Chain Monte Carlo simulation, widely used in Bayesian statistics, based on sampling from full conditional distributions for each variable or group of variables. The name comes from the method being first used on Gibbs random fields modeling of images by Geman and Geman (1984).

1
vote
1answer
109 views

How to Gibbs sample proportional to a probability

I am reading this tutorial on Hierarchical Chinese Restaurant Process. On pdf page 141 (slide title: MCMC Problem Specification for N-grams) it says: $$F(s_{1,k})=\frac{\alpha^{S'_1+s_{1,k}}}{(\alpha)...
2
votes
1answer
348 views

Gibbs sampling with constraints

I am reading tutorials on Gibbs sampling for partition sampling in Dirichlet Process (Chinese Restaurant Process), and have been struggling to understand the terminology used in the tutorials. To ...
8
votes
2answers
2k views

Rao-Blackwellization of Gibbs Sampler

I am currently estimating a stochastic volatility model with Markov Chain Monte Carlo methods. Thereby, I am implementing Gibbs and Metropolis sampling methods.Assuming I take the mean of the ...
7
votes
2answers
7k views

Generating samples from Gibbs sampling

I am quite new to sampling. I am doing Gibbs sampling for a Bayesian network. I am aware about the algorithm for the Gibbs sampling but there's one thing I am not able to understand. For example let'...
3
votes
0answers
120 views

When 2% of the Bayesian Model have not converged?

I have model with 20000 latent parameters, set up in a Gibb's sampler. 98% of the parameters and sometimes 99.5% of the parameters satisfy the Geweke convergence statistic, have low autocorrelation ...
4
votes
1answer
170 views

Gibbs sampling, what to use?

My question concerns Gibbs sampling. Suppose that I have three unknown quantities, $\mu, \sigma^2$ and $c$. I have given prior information and I have given the likelihood which allows me to compute ...
3
votes
1answer
604 views

Gibbs sampling and Conjugate Priors

Are conjugate priors required when performing Gibbs sampling?
4
votes
1answer
368 views

Gibbs sampling and conditional distribution

I need to simulate the posterior distribution of intraclass correlation coefficient $\pi(\rho|y)$ where $y$ is the data set and $\rho=\frac{\sigma_a^2}{\sigma_a^2+\sigma_e^2}$ with $\sigma^2_a\sim IG(\...
5
votes
0answers
606 views

Gibbs sampling deriving complete conditionals with mixture priors

My question is about the derivation of the complete conditionals for Gibbs sampling in a hierarchical model where some of the parameters are mixtures of point-masses and Normal distributions. The ...
1
vote
1answer
961 views

Gibbs sampling in R

Let X a continous variable and Y a binary variable with joint distribution : $$p(x,y;\beta,\rho_1,\rho_2,\phi_1,\phi_2)=\frac{1}{Z(\beta,\rho_1,\rho_2,\phi_1,\phi_2)}\exp(-0.5 \beta x^2+1_{y=0}\rho_1 ...
1
vote
1answer
72 views

Gibbs sampling for mixed variables [duplicate]

Let X a continous variable and Y a binary variable with joint distribution : $$p(x,y;\beta,\rho_1,\rho_2,\phi_1,\phi_2)=\frac{1}{Z(\beta,\rho_1,\rho_2,\phi_1,\phi_2)}\exp(-0.5 \beta x^2+1_{y=0}\rho_1 ...
2
votes
2answers
424 views

Gibbs sampling and mixed distribution

For a project, I need to simulate from a joint distribution with both continuous and discrete variables that are dependent. The conditional distribution of any variable given the rest is known. I ...
4
votes
2answers
359 views

Resources about Gibbs sampling in hybrid Bayesian networks

I've been trying to get my hands on a substantial resource for using Gibbs sampling in hybrid Bayesian networks, that is, networks with both continuous and discrete variables. So far I can't say I ...
5
votes
2answers
797 views

slice sampling within a Gibbs sampler

Questions My questions are: Is the following slice-sampling-within-Gibbs approach valid? Is there a good reference out there that uses, or better yet, justifies it? Context I'm trying to sample ...
4
votes
2answers
274 views

The Harris recurrence of a stepping-out slice-sampling-within-Gibbs MCMC

I want to use a multistage version of the MCMC here. That is, I want to use a Gibbs sampler to draw from a general joint distribution $p(x_1, x_2, x_3, \ldots)$ with a Gibbs step for each full ...
0
votes
0answers
55 views

How does GibbsLDA++ ensure that we are sampling from a good posterior?

This is an extending of this question, which asked that whether we should do some estimating to ensure that we are really using a likely topic assignment instead of the one happened with low ...
1
vote
0answers
791 views

How to check the convergence in the collapsed Gibbs sampling of LDA? [closed]

I am trying to implement the LDA model fit by collapsed Gibbs sampling by myself. I have go through this article. And there is a clear pseudo code (section 5.5), ...
2
votes
1answer
161 views

MCMC using GIBBS sampling: can different burn-in be used for different parameters?

I have run a stochastic volatility model with 4 parameters. I have used the Heidelberg and Welch convergence diagnostic. The result shows 3 out of 4 parameters have passed the stationary and half-...
2
votes
0answers
384 views

Why is the posterior the stationary distribution of a Gibbs chain?

I'm having trouble understanding the setup here. I'm following Probabilistic Graphical Models by Koller and Friedman. They say that we wish to generate samples from the posterior distribution $P(\...
7
votes
3answers
5k views

PyMC: how can I define a function of two stochastic variables, with no closed-form distribution?

I'm learning PyMC and basically I have a random variable $Z = X + Y$ where (say) $X \sim \mathrm{Normal}(\theta_X)$ and $Y \sim \mathrm{Lognormal}(\theta_Y)$ and $Z$ has no simple closed-form ...
2
votes
0answers
987 views

Deriving mean and variance of the posterior distribution

I have a simple linear model: $y_{i}=\mu+e_{i}$ for $i=1,...,n$, where $P(e_{i})=w\mathcal{N}(0,\sigma^2) + (1-w)\mathcal{N}(0,k^2\sigma^2)$ with $w=0.9$, $k=10$ and $\sigma=0.1$. It can be understood ...
0
votes
0answers
79 views

Uniform sampling of constrained binary vectors by Gibbs sampling

General statement of the problem: Let $x,y$ be two binary vectors, connected by the following constrains: $$y=f(x),\qquad x=g(y)$$ That is, $x$ determines $y$, and $y$ determines $x$. There are many ...
2
votes
2answers
117 views

How to estimate biases from coin and dice using only observed dice throws in this setup?

To help me understand some concepts I'm learning in my first exposition to machine learning, I'm trying to tackle the following "simple" problem The setup of the problem is as follows: My friend ...
3
votes
0answers
168 views

Derive Marginal Posterior to set up Gibbs-Sampler

I am currently trying to replicate a Hierarchical Model for multivariate returns proposed in the paper Portfolio selection using hierarchical Bayesian analysis and MCMC methods. However, in order to ...
3
votes
1answer
261 views

Sampling a random binary matrix with “Gaussian” probability distribution

Let $A_{ij}$ be a $n\times n$ random binary matrix with probability mass function $P(A)$ given by $$ \log P(A)=-\frac 12 \mathrm{tr}\left[\left(A-M\right)^TV\left(A-M\right)\right] + C, $$ where $M$ ...
1
vote
1answer
375 views

Collapsed Gibbs Sampling in Mixture Models

I tried to learn how Gibbs sampling works on Mixture models by studying David Blei's notes: http://www.cs.columbia.edu/~blei/fogm/2015F/notes/mixtures-and-gibbs.pdf In the equation 28: $p(z_i = k| ...
2
votes
0answers
97 views

If all components of a hierarchical model have not converged, can we say that any parameters have truly converged?

I'm working with a hierarchical regression model of the following form similar to that presented in Peter D. Hoff's book, A First Course in Bayesian Statistical Methods: $\boldsymbol{Y}_j \sim \text{...
4
votes
2answers
885 views

Is Slice Sampling a special case of Gibbs Sampling?

I read on this thread the following: If you can use both the gibbs sampler and slice sampling to sample from a posterior I would use the Gibbs sampler as the slice sampler seems unnecessary to ...
6
votes
2answers
1k views

Gibbs sampler gets stuck in local mode

I am very new to statistics and trying to implement a Gibbs sampler. However, according to wikipedia https://en.wikipedia.org/wiki/Gibbs_sampling and this discussion thread http://metaoptimize.com/qa/...
16
votes
1answer
3k views

Does the Gibbs Sampling algorithm guarantee detailed balance?

I have it on supreme authority1 that Gibbs Sampling is a special case of the Metropolis-Hastings algorithm for Markov Chain Monte Carlo sampling. The MH algorithm always gives a transition probability ...
1
vote
0answers
203 views

How to sample random variables (x,y) from a bivariate Cauchy distribution using a Gibbs sampler?

A bivariate Cauchy distribution is equivalent to a bivariate t-distribution with 1 degree of freedom.
1
vote
1answer
58 views

Deriving Gibbs sampler for specific mixture model

Let $\theta_i$ be an indicator which is $0$ if score, $X_i$, is the same for both opponents, $1$ if different: $X_i|\theta_i \stackrel{\text{ind}}{\sim} (1-\theta_i) U(0, 1) + \theta_i Beta(1, \...
1
vote
2answers
861 views

Implementing Latent Dirichlet Allocation - notation confusion

I am trying to implement LDA using the collapsed Gibbs sampler from http://www.uoguelph.ca/~wdarling/research/papers/TM.pdf the main algorithm is shown below I'm a bit confused about the notation ...
3
votes
2answers
583 views

Markov Chain Monte Carlo (MCMC) with transformed data

I want to obtain an estimate of a parameter $\Theta$ in a model for a random variable $X$ dependent on $\Theta$ with known but complicated likelihood $L(\Theta|X) = p(X|\Theta)$. $X$ is not directly ...
5
votes
1answer
846 views

Gibbs sampling from a complex full conditional

I have a sampling question relating to Gibbs sampling of a complicated full conditional. Supposed I have a complicated full conditional that I want a single sample from $p(\theta_i$|$\theta_{-i}$, $...
1
vote
0answers
85 views

How does one determine which variables can be collapsed in Gibbs Sampling?

I am going through the derivation for Gibbs Sampling update equations for LDA. The claim is that $\theta_{d}$ (document specific topic distribution) and $\phi_k$ the topic-word distribution can be ...
2
votes
1answer
105 views

Efficiently sampling from mixture distribution posterior

I have the following model: $$ \begin{align} \pi_1\sim & \text{Unif}(0,1)\\ \lambda_1,\lambda_2\sim & \text{Ga}(1,1)\\ z_i\sim & \pi_1^{1(z_i=1)}\pi_2^{1(z_i=2)}\\ p(y_i|\lambda_1,\...
1
vote
1answer
47 views

Is it possible for iterations to spike in Gibbs sampling?

After performing Gibbs sampling, I looked at a trace plot for one of my parameters and it appeared to spike at certain points. Is this possible or is it likely that I just coded my sampler wrong?
1
vote
1answer
206 views

Gibbs sampling for reducible chain

I am new to Gibbs sampling and I ran into a problem with irreducibility. For the Gibbs sampler to work the Markov chain has to be irreducible. But that assumption is not satisfied in my probability ...
4
votes
1answer
784 views

In Bayesian analysis, how to sample from full conditional given uniform prior and normal data likelihood?

[EDIT] This question comes from the example of OpenBUGS manual: Stagnant: a changepoint problem and an illustration of how NOT to do MCMC! I also asked another question regarding this example. [...
1
vote
1answer
87 views

Truncated prior leads to non-intuitive posterior

I am setting up a linear regression model for continuous data that is Normally distributed. For this model, I want to assume that my $\beta$ predictor is truncated to be positive, that is $$\beta \sim ...
0
votes
0answers
207 views

Is it possible for Metropolis sampling to converge to the wrong value?

I have simulated data under three parameters of interest, say a, b, c. The prior I put on c was a Gamma, so it only takes positive values. The full conditionals of a and b are known distributions, but ...
0
votes
1answer
175 views

Metropolis sampling (symmetric proposal distribution)

Can Metropolis sampling be used in conjunction with Gibbs sampling? So for example, if I have three parameters of interest, but only two of them have full conditionals that are known distributions, ...
1
vote
0answers
79 views

Gibbs within Collapsed Gibbs?

I have a model with variables $X_{1}, X_{2}, X_{3}, X_{4}$. I would like to sample it within a larger MCMC chain using: $(X_{1}, X_{2}) \sim P(X_{1}, X_{2})$ $(X_{3}, X_{4}) \sim P(X_{3}, X_{4} \mid ...
2
votes
0answers
401 views

Practical Implementation of Gibbs Sampling in Latent Dirichlet Allocation

In the collapsed Gibbs sampling version of LDA, the posterior distribution of topic assignments for each word is sampled. From what I have read (e.g. http://people.cs.umass.edu/~wallach/courses/s11/...
3
votes
1answer
963 views

Gibbs sampling with mixed prior using a Metropolis-Hastings step

My questions are about a sampling procedure for fitting a Bayesian hierarchical model where one of the priors is a mixture distribution of discrete and continuous parts. The model is not my own but I ...
2
votes
2answers
367 views

Gibbs sampler implementation

I am just getting started with the Gibbs Sampler and came across an implementation from here and here and here. All of theses implementations are based on the first article. There is an inner loop ...
0
votes
1answer
82 views

Gibbs sampling: ancillary and sufficient parametrization

After asking a question about Gibbs sampling earlier, I have another one for you. I have not been able to find laymen's background on this, the only referenced use I've found for this is in ...
3
votes
1answer
2k views

Gibbs sampling convergence

In an astronomical context, the authors of a paper desire to use a Gibbs algorithm. Please note: I am inexperience in MCMC algorithms, and specifically in Gibbs sampling. What we want, in essence, is ...
5
votes
2answers
3k views

How to reduce autocorrelation in Metropolis algorithm?

I've been using a Metropolis/Gibbs sampler combination to generate a joint density for some parameters(it is a hierarchical model, with $y_i\sim Poisson(\lambda_i)$, $\lambda_i\sim Gamma(\alpha,\beta)...