Questions tagged [gibbs]

The Gibbs sampler is a simple form of Markov Chain Monte Carlo simulation, widely used in Bayesian statistics, based on sampling from full conditional distributions for each variable or group of variables. The name comes from the method being first used on Gibbs random fields modeling of images by Geman and Geman (1984).

Filter by
Sorted by
Tagged with
1
vote
1answer
365 views

When can the collapsed Gibbs sampler be implemented?

I understand Gibbs sampling is a means of statistics inference, and it seems that sometimes certain variables can be integrated out in the sampling process, known as collapsed Gibbs sampling. I really ...
1
vote
1answer
274 views

Expected value of an estimate done using Gibbs sampling

This is related to this question. I am concerned about the expected value of a function estimated using sampling. If the samples were obtained by a sampling method such as Gibbs sampling, and if I ...
1
vote
1answer
204 views

MCMC sampling with sum constraints

I'm interested in sampling a collection of variables with a sum constraint on them. For a simplified example: Prior: $X \sim \mathcal{N}(0, 1)$ $Y \sim \mathcal{N}(0, 1)$ Observation: $X + Y = 1$ ...
1
vote
2answers
917 views

Implementing Latent Dirichlet Allocation - notation confusion

I am trying to implement LDA using the collapsed Gibbs sampler from http://www.uoguelph.ca/~wdarling/research/papers/TM.pdf the main algorithm is shown below I'm a bit confused about the notation ...
1
vote
2answers
158 views

throwing away all Gibbs samples after approximation

This is more of a theory question, consider: $$P(w_1|D)=\int P(w_1|S)P(S|D)d(S)$$ which we approximate via Gibbs sampling $S$ (assume the initial state of the Gibbs sampler is denoted by $M_0$), ...
1
vote
1answer
1k views

Gibbs sampling in R

Let X a continous variable and Y a binary variable with joint distribution : $$p(x,y;\beta,\rho_1,\rho_2,\phi_1,\phi_2)=\frac{1}{Z(\beta,\rho_1,\rho_2,\phi_1,\phi_2)}\exp(-0.5 \beta x^2+1_{y=0}\rho_1 ...
1
vote
1answer
102 views

Truncated prior leads to non-intuitive posterior

I am setting up a linear regression model for continuous data that is Normally distributed. For this model, I want to assume that my $\beta$ predictor is truncated to be positive, that is $$\beta \sim ...
1
vote
1answer
869 views

How to incorporate parameter constraints in Metropolis Hastings

I am working on parameter estimation of GARCH model with Metropolis Hastings. But the results I have got doesn't look reasonable, actually it is quite different from what I have got from Gibbs sampler....
1
vote
1answer
2k views

Metropolis-Hastings using log of the density

Does Metropolis-Hastings work with the log of the proposal and the density to be sampled from? That is, say we want to sample from a density $\pi(x)$, using a proposal $q(x|x^{old})$, will the ...
1
vote
2answers
1k views

Unsupervised Bayesian naive Bayes

I'm reading a paper Gibbs sampling for the uninitiated. In this paper, the authors try to use Gibbs sampling for a bayesian naive bayes model. They formalize the model as a graphical model in page 8. ...
1
vote
1answer
2k views

Need help on Gibbs sampling with truncated normal and gamma

I am trying to use Gibbs Sampling to simulate a random sample from a joint distribution $f(\beta ,{{Z}_{1}},...,{{Z}_{75}},{{\lambda }_{1}},...,{{\lambda }_{75}})$, where the fully conditioned ...
1
vote
1answer
554 views

Posterior expression for Gibbs sampling

I am trying to estimate parameters of a two dimensional Normal distribution using Gibbs sampling. While it was very easy transform the posterior equation for mean vector to a single dimension normal ...
1
vote
1answer
444 views

which R package is good for gibbs sampler when the likelihood function is complex

I see a lot of examples using MCMC to solve for posterior distribution when the likelihood is simply one of linear regression. What if the likelihood is an ugly, complex function. Which R package can ...
1
vote
1answer
92 views

Gibbs Sampler Consistency

Simple question that I haven't found easily online: why are the estimates obtained from a Gibbs sampler consistent (converge to the true probabilities)?
1
vote
1answer
619 views

How to do MC integration from gibbs sampling of posterior

I'm a beginner of MCMC. Two questions confused me. If I know the posterior distribution, and from the Gibbs sampling, I got the sampled parameter, so How to draw the histogram with y axis as ...
1
vote
1answer
248 views

Conditional posterior probability density (steps)

I'm trying to understand how to condition a probabilistic posterior distribution. Consider the following probability density: $$ p(\alpha, \beta | y) = \prod_{i=1}^n (\alpha+\beta t_i)^{y_i}e^{-(\...
1
vote
1answer
418 views

Collapsed Gibbs Sampling in Mixture Models

I tried to learn how Gibbs sampling works on Mixture models by studying David Blei's notes: http://www.cs.columbia.edu/~blei/fogm/2015F/notes/mixtures-and-gibbs.pdf In the equation 28: $p(z_i = k| ...
1
vote
1answer
85 views

Gibbs Sampling Inserting Some Known Predictors

Imagine you would like to use a simple Gibbs sampling to resample from a joint probability distribution which is difficult to model (but you know all the conditionals $Pr\left(X_i|X_1,...,X_{i-1},X_{i+...
1
vote
1answer
747 views

Gibbs Sampling for Gaussian Mixtures

Does the Gibbs sampler converge to a global maximum in the presence of multiple modes? For example in case of a Gaussian mixture distribution?
1
vote
2answers
1k views

Sampling covariance matrix using Gibbs sampling

I am sampling covariance matrix from a Inverse Wishart distribution. In one dimensional case, after doing sufficient iterations I am taking the mode value for variance (after removing the burn-in ...
1
vote
1answer
29 views

Gibbs Sampling - Calculating the full conditionals from the joint density

Given a joint density, $f(x_1, x_2)$, can its pmf/pdf be found generally by the method outlined below: For a joint density, $f(x_1, x_2)$ if we hold $x_2$ constant in the joint density, we will get ...
1
vote
1answer
41 views

What is the role of simulated annealing in Gibbs sampling?

While I was reading about Gibbs sampling, I happened to see "simulated annealing" but what is it doing in Gibbs sampling? Although I don't understand the full context of simulated annealing, I am ...
1
vote
1answer
183 views

Conditional distribution in this Gaussian Mixture Model

Say I observe $N$ observations $\{x_1, \dots, x_N\}$ from a $k$ component Gaussian Mixture model, with $k > 0$ known and is such that each $x_i|\boldsymbol{\pi}, \boldsymbol{\mu} \sim \sum_{j=1}^{k}...
1
vote
1answer
155 views

Gibbs sampler for Dirichlet Process concentration parameter

I am trying to implement a Gibbs sampler for Hierarchical Dirichlet process, but I cannot seem to correctly estimate the concentration parameters. I therefore started testing just this part of a ...
1
vote
1answer
697 views

deriving posterior conditionals for gibbs sampling

I'm new to Bayesian inference and Gibbs sampling in general, and I'm struggling trying to derive the conditional posteriors for a particular data generating process I'm trying to model. The model I ...
1
vote
1answer
225 views

Sampling from a posterior with Gibbs sampling

In an image processing class, I dont really get behind the idea how to 'sample from a posterior' with Gibbs sampling. We have a posterior distribution: $f(z_1, .. ,z_n \mid x_1,.. ,x_n) := f(z \mid x)...
1
vote
1answer
583 views

Influence of word counts from DTM on LDA with Gibbs Sampling

I'm trying to wrap my head around Topic Modeling based on LDA with Gibbs sampling (Griffiths, Steyvers 2004: Finding Scientific Topics). What struck me when reading some Python implementations like ...
1
vote
1answer
558 views

Sampling from the joint distribution p(x,y) when y = f(x)

Suppose I want to sample from the joint distribution $p(X, Y)$, where $X$ is a random variable and $Y = f(X)$ where $f$ is a known function of $X$. However, sampling from $p(X,Y)$ directly is hard. ...
1
vote
1answer
469 views

Why Gibbs sampling?

I am new to Gibbs sampling and sampling in general, so here is a basic question. I am reading this tutorial. Equation (40) is our complicated joint probability and equation (49) the less complicated ...
1
vote
1answer
521 views

Generating Sample Path of HMM via Gibbs Sampling

I have a question regarding section 7.1.1 here. I have two questions to ask you. What does the following sentence mean? we shall be drawing values for $C_T,C_{T-1},\cdots,C_1$ in order. How the ...
1
vote
1answer
120 views

How to Gibbs sample proportional to a probability

I am reading this tutorial on Hierarchical Chinese Restaurant Process. On pdf page 141 (slide title: MCMC Problem Specification for N-grams) it says: $$F(s_{1,k})=\frac{\alpha^{S'_1+s_{1,k}}}{(\alpha)...
1
vote
1answer
77 views

Gibbs sampling for mixed variables [duplicate]

Let X a continous variable and Y a binary variable with joint distribution : $$p(x,y;\beta,\rho_1,\rho_2,\phi_1,\phi_2)=\frac{1}{Z(\beta,\rho_1,\rho_2,\phi_1,\phi_2)}\exp(-0.5 \beta x^2+1_{y=0}\rho_1 ...
1
vote
1answer
66 views

Deriving Gibbs sampler for specific mixture model

Let $\theta_i$ be an indicator which is $0$ if score, $X_i$, is the same for both opponents, $1$ if different: $X_i|\theta_i \stackrel{\text{ind}}{\sim} (1-\theta_i) U(0, 1) + \theta_i Beta(1, \...
1
vote
1answer
53 views

Is it possible for iterations to spike in Gibbs sampling?

After performing Gibbs sampling, I looked at a trace plot for one of my parameters and it appeared to spike at certain points. Is this possible or is it likely that I just coded my sampler wrong?
1
vote
1answer
108 views

PyMC consistently under estimating results found in paper. Possibly not sampling enough?

I have been trying to build confidence in (my ability to correctly use) PyMC by working examples. Namely, I have been working on Chickering and Pearl 1997, and more specifically on their 'artificial' ...
1
vote
2answers
678 views

Any good book for learning probability programming

Are there any good books for me to learn probability programming? For example, I am new to Latent Dirichlet allocation (LDA) and Gibbs sampling. I have read some books about the techniques, but it ...
1
vote
1answer
506 views

Computing conditional expectation of ordered normal random variables

There are $m$ normally distributed, independent random variables $N_1, \ldots, N_m$ with distinct means $\mu_1, \ldots \mu_m$ and standard deviations $\sigma_1, \ldots, \sigma_m$. Then, we observe a ...
1
vote
0answers
28 views

Sampling states of an “unnatural” Hamiltonian System

I would like to sample from a Gibbs distribution given by $$f(p, q) = \frac{1}{\mathcal{Z}}e^{-H(p, q; \omega, J)}$$ where $H$ is the Hamiltonian on generalized coordinates $(p,q)\in \mathbb{R}^{2n}...
1
vote
0answers
21 views

Why the nodes in a Boltzmann machine need to be sampled one at a time?

Typically, we use Gibbs sampling to update (or generate samples from) energy based models. This means we update each node while keeping its markov blanket constant. Why can't we update/sample all ...
1
vote
0answers
36 views

Bayesian Gamma Regression Update

I'm looking for a resource that explains how to do update the coefficients for a Bayesian gamma regression using Gibbs sampling. Specifically, if $y_i \sim Gamma(\alpha,\beta_i)$ and my data ...
1
vote
0answers
17 views

Markov chain Monte Carlo, Mixing Time

How do you estimate the mixing time for a markov chain? I read somewhere one can use the sum of the auto-correlation coefficients or the sum of the auto-covariance coefficients, but I cannot seem to ...
1
vote
0answers
45 views

What's the point of Gibbs Sampling? [duplicate]

I am reading a book on doing Bayesian Data Analysis. I have just learned what the Metropolis Hastings (MH) Algo does, at least in relation to Bayesian Data Analysis. My understanding of the MH Algo ...
1
vote
1answer
24 views

Assign an error to the parameters of MAP estimate

Through a MCMC Gibbs sampler I obtain $M$ chains of the parameters vector $\mathbf{\theta}$, meaning that each component of $\mathbf{\theta}$ is the value of one parameter at a given iteration. ...
1
vote
0answers
25 views

Is there something wrong with my Bayesian hierarchical spatio-temporal model?

I built a Bayesian spatio-temporal model and one of the parameters to be estimated is the random spatial effects s. The random spatial effect is assigned an intrinsic conditional autoregressive prior (...
1
vote
0answers
39 views

Understand the Holmes and Held (2006) Bayesian probit MCMC algorithm

Holmes and Held (2006) suggest a simple approach to reduce autocorrelation in the MCMC algorithm proposed by Albert and Chib (1993). HH (2006) propose to update $\beta$ and $z$ jointly, making use of ...
1
vote
0answers
31 views

Finding the posterior distribution of mean and variance given data sample using Gibbs Sampling?

I have the following hierachical bayesian model - $\mathbf{x}|\mathbf{c},\sigma^2 \sim \mathcal{N}(\mathbf{x}|\mathbf{c},\sigma^2)$ $\mathbf{c}|\mathbf{c}_1,\sigma^2_2 \sim \mathcal{N}(\mathbf{c}|\...
1
vote
0answers
24 views

Gibbs Sampler for mixture models: shall I skip some samples to avoid to use correlated samples? [duplicate]

I am implementing a Gibbs sampler in order to estimate the parameters of a mixture model. Assuming that the parameters are contained in a vector $\theta$ what I will do is: Implement and run the ...
1
vote
0answers
44 views

How exactly does Gibbs sampling work in Markov Networks?

I was going through the Probabilistic Graphical Modelling course by Stanford and they used a network such as this one-https://imgur.com/gallery/k0C8FY2 Now if we want to sample P(A|B), how would we ...
1
vote
0answers
52 views

HDP: Gibbs sampler implementation

I am trying to recreate the model proposed by Gao et al. (2011), based on the Hierarchical Dirichlet Process proposed by Teh and al. (2005). To estimate the model (let's call it iHDP) I need to ...
1
vote
0answers
59 views

Non-Identifiable Multivariate Normal Posterior

So I have a theoretical question about what looks like, in my opinion, a multivariate normal distribution. The issue comes with the fact that the data is distributed with likelihood: Y |θ1, θ2 ∼ N(θ1 ...

1 3 4 5 6 7