Questions tagged [gibbs]

The Gibbs sampler is a simple form of Markov Chain Monte Carlo simulation, widely used in Bayesian statistics, based on sampling from full conditional distributions for each variable or group of variables. The name comes from the method being first used on Gibbs random fields modeling of images by Geman and Geman (1984).

Filter by
Sorted by
Tagged with
1
vote
1answer
91 views

Truncated prior leads to non-intuitive posterior

I am setting up a linear regression model for continuous data that is Normally distributed. For this model, I want to assume that my $\beta$ predictor is truncated to be positive, that is $$\beta \sim ...
1
vote
1answer
801 views

How to incorporate parameter constraints in Metropolis Hastings

I am working on parameter estimation of GARCH model with Metropolis Hastings. But the results I have got doesn't look reasonable, actually it is quite different from what I have got from Gibbs sampler....
1
vote
1answer
2k views

Metropolis-Hastings using log of the density

Does Metropolis-Hastings work with the log of the proposal and the density to be sampled from? That is, say we want to sample from a density $\pi(x)$, using a proposal $q(x|x^{old})$, will the ...
1
vote
2answers
1k views

Unsupervised Bayesian naive Bayes

I'm reading a paper Gibbs sampling for the uninitiated. In this paper, the authors try to use Gibbs sampling for a bayesian naive bayes model. They formalize the model as a graphical model in page 8. ...
1
vote
1answer
2k views

Need help on Gibbs sampling with truncated normal and gamma

I am trying to use Gibbs Sampling to simulate a random sample from a joint distribution $f(\beta ,{{Z}_{1}},...,{{Z}_{75}},{{\lambda }_{1}},...,{{\lambda }_{75}})$, where the fully conditioned ...
1
vote
1answer
551 views

Posterior expression for Gibbs sampling

I am trying to estimate parameters of a two dimensional Normal distribution using Gibbs sampling. While it was very easy transform the posterior equation for mean vector to a single dimension normal ...
1
vote
1answer
382 views

which R package is good for gibbs sampler when the likelihood function is complex

I see a lot of examples using MCMC to solve for posterior distribution when the likelihood is simply one of linear regression. What if the likelihood is an ugly, complex function. Which R package can ...
1
vote
1answer
88 views

Gibbs Sampler Consistency

Simple question that I haven't found easily online: why are the estimates obtained from a Gibbs sampler consistent (converge to the true probabilities)?
1
vote
1answer
398 views

Collapsed Gibbs Sampling in Mixture Models

I tried to learn how Gibbs sampling works on Mixture models by studying David Blei's notes: http://www.cs.columbia.edu/~blei/fogm/2015F/notes/mixtures-and-gibbs.pdf In the equation 28: $p(z_i = k| ...
1
vote
1answer
84 views

Gibbs Sampling Inserting Some Known Predictors

Imagine you would like to use a simple Gibbs sampling to resample from a joint probability distribution which is difficult to model (but you know all the conditionals $Pr\left(X_i|X_1,...,X_{i-1},X_{i+...
1
vote
1answer
726 views

Gibbs Sampling for Gaussian Mixtures

Does the Gibbs sampler converge to a global maximum in the presence of multiple modes? For example in case of a Gaussian mixture distribution?
1
vote
2answers
1k views

Sampling covariance matrix using Gibbs sampling

I am sampling covariance matrix from a Inverse Wishart distribution. In one dimensional case, after doing sufficient iterations I am taking the mode value for variance (after removing the burn-in ...
1
vote
1answer
129 views

Conditional distribution in this Gaussian Mixture Model

Say I observe $N$ observations $\{x_1, \dots, x_N\}$ from a $k$ component Gaussian Mixture model, with $k > 0$ known and is such that each $x_i|\boldsymbol{\pi}, \boldsymbol{\mu} \sim \sum_{j=1}^{k}...
1
vote
1answer
114 views

Gibbs sampler for Dirichlet Process concentration parameter

I am trying to implement a Gibbs sampler for Hierarchical Dirichlet process, but I cannot seem to correctly estimate the concentration parameters. I therefore started testing just this part of a ...
1
vote
1answer
609 views

deriving posterior conditionals for gibbs sampling

I'm new to Bayesian inference and Gibbs sampling in general, and I'm struggling trying to derive the conditional posteriors for a particular data generating process I'm trying to model. The model I ...
1
vote
1answer
210 views

Sampling from a posterior with Gibbs sampling

In an image processing class, I dont really get behind the idea how to 'sample from a posterior' with Gibbs sampling. We have a posterior distribution: $f(z_1, .. ,z_n \mid x_1,.. ,x_n) := f(z \mid x)...
1
vote
1answer
539 views

Influence of word counts from DTM on LDA with Gibbs Sampling

I'm trying to wrap my head around Topic Modeling based on LDA with Gibbs sampling (Griffiths, Steyvers 2004: Finding Scientific Topics). What struck me when reading some Python implementations like ...
1
vote
1answer
409 views

Sampling from the joint distribution p(x,y) when y = f(x)

Suppose I want to sample from the joint distribution $p(X, Y)$, where $X$ is a random variable and $Y = f(X)$ where $f$ is a known function of $X$. However, sampling from $p(X,Y)$ directly is hard. ...
1
vote
1answer
459 views

Why Gibbs sampling?

I am new to Gibbs sampling and sampling in general, so here is a basic question. I am reading this tutorial. Equation (40) is our complicated joint probability and equation (49) the less complicated ...
1
vote
1answer
570 views

How to do MC integration from gibbs sampling of posterior

I'm a beginner of MCMC. Two questions confused me. If I know the posterior distribution, and from the Gibbs sampling, I got the sampled parameter, so How to draw the histogram with y axis as ...
1
vote
1answer
497 views

Generating Sample Path of HMM via Gibbs Sampling

I have a question regarding section 7.1.1 here. I have two questions to ask you. What does the following sentence mean? we shall be drawing values for $C_T,C_{T-1},\cdots,C_1$ in order. How the ...
1
vote
1answer
202 views

Conditional posterior probability density (steps)

I'm trying to understand how to condition a probabilistic posterior distribution. Consider the following probability density: $$ p(\alpha, \beta | y) = \prod_{i=1}^n (\alpha+\beta t_i)^{y_i}e^{-(\...
1
vote
1answer
113 views

How to Gibbs sample proportional to a probability

I am reading this tutorial on Hierarchical Chinese Restaurant Process. On pdf page 141 (slide title: MCMC Problem Specification for N-grams) it says: $$F(s_{1,k})=\frac{\alpha^{S'_1+s_{1,k}}}{(\alpha)...
1
vote
1answer
75 views

Gibbs sampling for mixed variables [duplicate]

Let X a continous variable and Y a binary variable with joint distribution : $$p(x,y;\beta,\rho_1,\rho_2,\phi_1,\phi_2)=\frac{1}{Z(\beta,\rho_1,\rho_2,\phi_1,\phi_2)}\exp(-0.5 \beta x^2+1_{y=0}\rho_1 ...
1
vote
1answer
59 views

Deriving Gibbs sampler for specific mixture model

Let $\theta_i$ be an indicator which is $0$ if score, $X_i$, is the same for both opponents, $1$ if different: $X_i|\theta_i \stackrel{\text{ind}}{\sim} (1-\theta_i) U(0, 1) + \theta_i Beta(1, \...
1
vote
1answer
50 views

Is it possible for iterations to spike in Gibbs sampling?

After performing Gibbs sampling, I looked at a trace plot for one of my parameters and it appeared to spike at certain points. Is this possible or is it likely that I just coded my sampler wrong?
1
vote
1answer
105 views

PyMC consistently under estimating results found in paper. Possibly not sampling enough?

I have been trying to build confidence in (my ability to correctly use) PyMC by working examples. Namely, I have been working on Chickering and Pearl 1997, and more specifically on their 'artificial' ...
1
vote
1answer
603 views

Gibbs Sampling Detecting Change point in time series

I was reading through this one page paper on using Gibbs sampling for detecting a change point in a time series like data. While I understand the part where the $\lambda$ and $\phi$ are chosen from a ...
1
vote
2answers
626 views

Any good book for learning probability programming

Are there any good books for me to learn probability programming? For example, I am new to Latent Dirichlet allocation (LDA) and Gibbs sampling. I have read some books about the techniques, but it ...
1
vote
1answer
500 views

Computing conditional expectation of ordered normal random variables

There are $m$ normally distributed, independent random variables $N_1, \ldots, N_m$ with distinct means $\mu_1, \ldots \mu_m$ and standard deviations $\sigma_1, \ldots, \sigma_m$. Then, we observe a ...
1
vote
1answer
23 views

Assign an error to the parameters of MAP estimate

Through a MCMC Gibbs sampler I obtain $M$ chains of the parameters vector $\mathbf{\theta}$, meaning that each component of $\mathbf{\theta}$ is the value of one parameter at a given iteration. ...
1
vote
0answers
19 views

Is there something wrong with my Bayesian hierarchical spatio-temporal model?

I built a Bayesian spatio-temporal model and one of the parameters to be estimated is the random spatial effects s. The random spatial effect is assigned an intrinsic conditional autoregressive prior (...
1
vote
0answers
33 views

Understand the Holmes and Held (2006) Bayesian probit MCMC algorithm

Holmes and Held (2006) suggest a simple approach to reduce autocorrelation in the MCMC algorithm proposed by Albert and Chib (1993). HH (2006) propose to update $\beta$ and $z$ jointly, making use of ...
1
vote
0answers
28 views

Finding the posterior distribution of mean and variance given data sample using Gibbs Sampling?

I have the following hierachical bayesian model - $\mathbf{x}|\mathbf{c},\sigma^2 \sim \mathcal{N}(\mathbf{x}|\mathbf{c},\sigma^2)$ $\mathbf{c}|\mathbf{c}_1,\sigma^2_2 \sim \mathcal{N}(\mathbf{c}|\...
1
vote
0answers
23 views

Gibbs Sampler for mixture models: shall I skip some samples to avoid to use correlated samples? [duplicate]

I am implementing a Gibbs sampler in order to estimate the parameters of a mixture model. Assuming that the parameters are contained in a vector $\theta$ what I will do is: Implement and run the ...
1
vote
0answers
36 views

How exactly does Gibbs sampling work in Markov Networks?

I was going through the Probabilistic Graphical Modelling course by Stanford and they used a network such as this one-https://imgur.com/gallery/k0C8FY2 Now if we want to sample P(A|B), how would we ...
1
vote
0answers
44 views

HDP: Gibbs sampler implementation

I am trying to recreate the model proposed by Gao et al. (2011), based on the Hierarchical Dirichlet Process proposed by Teh and al. (2005). To estimate the model (let's call it iHDP) I need to ...
1
vote
0answers
56 views

Non-Identifiable Multivariate Normal Posterior

So I have a theoretical question about what looks like, in my opinion, a multivariate normal distribution. The issue comes with the fact that the data is distributed with likelihood: Y |θ1, θ2 ∼ N(θ1 ...
1
vote
0answers
56 views

help interpreting plot of MCMC sample

I am estimating a model using MCMC (Gibbs Sampling). Because of the complexity of the model, I have been running two chains with many iterations. A plot of the draws for each parameter reveals a ...
1
vote
0answers
91 views

How to know whether a Gibbs sampler is irreducible? [duplicate]

How to know whether a Gibbs sampler is irreducible? I know that the Gibbs sampler in e.g. two variable case constructs a sequence of r.v.s $(X_1^{(i)}, X_2^{(i)})$ by sampling from the related ...
1
vote
0answers
19 views

Averaging Across Gibbs Sampling Runs with Reduced Dimensions

I need help thinking through my approach to Gibbs Sampling of many parameters and I'd like to know if there is literature on this topic: I have a dataset with 3 dimensions: ...
1
vote
0answers
30 views

Gibbs sampling with expectations instead of sampling

I see there is something called Iterated Conditional Models (ICM), which is a sort of Gibbs sampling where, instead of sampling, we use the value that maximizes the conditional. That is: ...
1
vote
0answers
232 views

Collapsed Gibbs sampler on Hierarchical Dirichlet Process Mixture Model

I am trying to design a collapsed Gibbs sampler on a mixture model based on Hierarchical Dirichlet Process ($g\sim DP(\gamma, b)$, $\pi\sim DP(\alpha, g)$ ). Should I resample from the posterior of ...
1
vote
0answers
191 views

Get different results with different sampling order in Gibbs sampling: what could be wrong?

In sampling a complex spatio-temporal model by Gibbs sampling, I found if I change the order of sampling (for example, to sample $P(\theta_1,\theta_2|D)$, in one try, I sample $\theta_1\sim P(\theta_1|...
1
vote
0answers
113 views

Gibbs Sampling / Monte Carlo with Weights

Consider pairs of data and their population weights $(y_i, \omega_i), i = 1, 2, \dots$ alongside some hierarchical structure, $$y_i\leftarrow\theta_{i(j)} \leftarrow \gamma$$ where $i(j)$ is perhaps ...
1
vote
1answer
55 views

Gibbs sampler -transformation of conditional posterior

If my conditional posterior $\pi(\sigma^{-2}|\mathbf{y },\mu)\sim Gamma(a,b)$, how can I get the conditional posterior $\pi(\sigma^{2}|\mathbf{y },\mu)$ with a transformation? The reason I ask is ...
1
vote
0answers
27 views

Sampling from a truncated random effects distribution

How would one sample observations $T_{ij} = U_i + \varepsilon_{ij}$, where the distribution of $U_i, \varepsilon_{ij}$ are known and mutually independent, condition on the fact that $L_{ij} \le T_{ij} ...
1
vote
0answers
62 views

Methods to solve a compound distribution MLE I can't write in closed form

What are some methods of solving a MLE parameter estimate where the likelihood function can't be written in closed form? I have a compound distribution that I don't have a closed form for (no ...
1
vote
0answers
49 views

Gibbs sampling confusion

I'm just wondering if i'm doing this process correctly i'm a little confused as to the answers i'm getting: I have $P(x=0,y=0) = P(x=1,y=1) = 0.5$ and $P(x=0,y=1) = P(x=1,y=0) = 0$ I calculated $P(...
1
vote
1answer
141 views

How does this Sampler work for the Concentration parameter of Dirichlet Process?

I am puzzled by how this Gibbs sampler on section 6 of Escobar & West (1995) works. To put it in simple words, the aim is to sample $\alpha$. The defined terms are: $$\eta\sim \texttt{Beta}(a,b)$$ ...