The Gibbs sampler is a simple form of Markov Chain Monte Carlo simulation, widely used in Bayesian statistics, based on sampling from full conditional distributions for each variable.

learn more… | top users | synonyms

0
votes
1answer
40 views

Gibbs sampling in R

Let X a continous variable and Y a binary variable with joint distribution : $$p(x,y;\beta,\rho_1,\rho_2,\phi_1,\phi_2)=\frac{1}{Z(\beta,\rho_1,\rho_2,\phi_1,\phi_2)}\exp(-0.5 \beta x^2+1_{y=0}\rho_1 ...
0
votes
1answer
30 views

Gibbs sampling for mixed variables [duplicate]

Let X a continous variable and Y a binary variable with joint distribution : $$p(x,y;\beta,\rho_1,\rho_2,\phi_1,\phi_2)=\frac{1}{Z(\beta,\rho_1,\rho_2,\phi_1,\phi_2)}\exp(-0.5 \beta x^2+1_{y=0}\rho_1 ...
1
vote
2answers
50 views

Gibbs sampling and mixed distribution

For a project, I need to simulate from a joint distribution with both continuous and discrete variables that are dependent. The conditional distribution of any variable given the rest is known. I ...
0
votes
0answers
13 views

Conditional Distribution of Hidden Markov Model

I am trying to implement a Gibbs sampling algorithm for a toy Hidden Markov Model, but I am having trouble deriving the target conditional distribution. I am generating data through the following ...
0
votes
0answers
7 views

R: dlmGibbsDIG observation variance

I am working with the dlmGibbsDIG function (dlm library) and am having some problems with understanding the output of the formula. I analyzed a stock return (dY) and 4 independent variables (dX) with ...
0
votes
0answers
8 views

How to determine and use the sampling lag in the collapsed Gibbs sampler?

I am implementing a collapsed Gibbs sampler for LDA model. According to this technical note's word, average a number of samples, and often it is desirable to leave an interval of L iteration ...
0
votes
0answers
15 views

How does GibbsLDA++ ensure that we are sampling from a good posterior?

This is an extending of this question, which asked that whether we should do some estimating to ensure that we are really using a likely topic assignment instead of the one happened with low ...
1
vote
0answers
74 views

How to check the convergence in the collapsed Gibbs sampling of LDA? [closed]

I am trying to implement the LDA model fit by collapsed Gibbs sampling by myself. I have go through this article. And there is a clear pseudo code (section 5.5), ...
5
votes
0answers
25 views

Dirichlet process mixture MCMC

I'm reading Markov Chain Sampling Methods for Dirichlet Process Mixture Models by Radford M. Neal. Equation (3.6) states that If $c=c_{j}$ for some $j\neq i $: $P\left(c_{i}=c\;|\;c_{-i}, y_{i}, ...
3
votes
1answer
30 views

MCMC using GIBBS sampling: can different burn-in be used for different parameters?

I have run a stochastic volatility model with 4 parameters. I have used the Heidelberg and Welch convergence diagnostic. The result shows 3 out of 4 parameters have passed the stationary and ...
3
votes
2answers
117 views

The Harris recurrence of a stepping-out slice-sampling-within-Gibbs MCMC

I want to use a multistage version of the MCMC here. That is, I want to use a Gibbs sampler to draw from a general joint distribution $p(x_1, x_2, x_3, \ldots)$ with a Gibbs step for each full ...
2
votes
0answers
54 views

Deriving mean and variance of the posterior distribution

I have a simple linear model: $y_{i}=\mu+e_{i}$ for $i=1,...,n$, where $P(e_{i})=w\mathcal{N}(0,\sigma^2) + (1-w)\mathcal{N}(0,k^2\sigma^2)$ with $w=0.9$, $k=10$ and $\sigma=0.1$. It can be understood ...
0
votes
0answers
26 views

Uniform sampling of constrained binary vectors by Gibbs sampling

General statement of the problem: Let $x,y$ be two binary vectors, connected by the following constrains: $$y=f(x),\qquad x=g(y)$$ That is, $x$ determines $y$, and $y$ determines $x$. There are many ...
0
votes
0answers
16 views

Gibbs Sampling for Multivariate Case

I understand the procedure for Gibbs sampling for bivariate case but I got confused about Gibbs sampler for multivariate normal distribution. I guess I should try to determine the conditional ...
0
votes
0answers
15 views

Derive Marginal Posterior to set up Gibbs-Sampler

I am currently trying to replicate a Hierarchical Model for multivariate returns proposed in the paper Portfolio selection using hierarchical Bayesian analysis and MCMC methods. However, in order to ...
2
votes
2answers
33 views

How to estimate biases from coin and dice using only observed dice throws in this setup?

To help me understand some concepts I'm learning in my first exposition to machine learning, I'm trying to tackle the following "simple" problem The setup of the problem is as follows: My friend ...
0
votes
0answers
18 views

Using two different methods for training and inferring

I have a model which is intractable to take derivatives wrt parameters and estimate them based on maximum likelihood. Even with an deterministic approximation like variational inference this is ...
0
votes
0answers
13 views

Independence yields exchange of summation and product in Gibbs Distribution

I am looking at the moment into statistical learning theory and encountered today the maximum entropy inference for k-mean clustering with a cost function given by: $$R(c)= \sum\limits_i || x_i - ...
3
votes
1answer
152 views

Sampling a random binary matrix with “Gaussian” probability distribution

Let $A_{ij}$ be a $n\times n$ random binary matrix with probability mass function $P(A)$ given by $$ \log P(A)=-\frac 12 \mathrm{tr}\left[\left(A-M\right)^TV\left(A-M\right)\right] + C, $$ where $M$ ...
1
vote
0answers
40 views

If all components of a hierarchical model have not converged, can we say that any parameters have truly converged?

I'm working with a hierarchical regression model of the following form similar to that presented in Peter D. Hoff's book, A First Course in Bayesian Statistical Methods: $\boldsymbol{Y}_j \sim ...
1
vote
0answers
23 views

Blocked Gibbs Sampling using Forward / Backward Algorithm

I am new to machine learning and have been reading about gibbs sampling. From my understanding, gibbs samples a single variable iteratively conditioned on all other variables. In blocked gibbs ...
0
votes
0answers
27 views

lasso gibbs sampler

Hi guys I tried to build a gibbs sampler for lasso regression. here is the model $y = Normal(x\beta,\sigma)\\ \beta = Laplace(alpha)\\ \sigma = Gamma(a,b)\\ \alpha = Gamma(c,d) $ I just used a ...
1
vote
1answer
34 views

Collapsed Gibbs Sampling in Mixture Models

I tried to learn how Gibbs sampling works on Mixture models by studying David Blei's notes: http://www.cs.columbia.edu/~blei/fogm/2015F/notes/mixtures-and-gibbs.pdf In the equation 28: $p(z_i = k| ...
4
votes
2answers
95 views

slice sampling within a Gibbs sampler

Questions My questions are: Is the following slice-sampling-within-Gibbs approach valid? Is there a good reference out there that uses, or better yet, justifies it? Context I'm trying to sample ...
2
votes
2answers
123 views

Gibbs sampler gets stuck in local mode

I am very new to statistics and trying to implement a Gibbs sampler. However, according to wikipedia https://en.wikipedia.org/wiki/Gibbs_sampling and this discussion thread ...
0
votes
0answers
54 views

Why sampling for inference of a new document in LDA?

Given a standard LDA model with few 1000 topics and few millions of documents, trained with Mallet / collapsed Gibbs sampler: When inferring a new document: Why not just skip sampling and simply use ...
1
vote
0answers
32 views

How to sample random variables (x,y) from a bivariate Cauchy distribution using a Gibbs sampler?

A bivariate Cauchy distribution is equivalent to a bivariate t-distribution with 1 degree of freedom.
7
votes
0answers
118 views

How to test if a cross-covariance matrix is non-zero

The background of my study: In a Gibbs sampling where we sample $X$ (the variable of interests) and $Y$ from $P(X|Y)$ and $P(Y|X)$ respectively, where $X$ and $Y$ are $k$-dimensional random vectors. ...
1
vote
1answer
25 views

Deriving Gibbs sampler for specific mixture model

Let $\theta_i$ be an indicator which is $0$ if score, $X_i$, is the same for both opponents, $1$ if different: $X_i|\theta_i \stackrel{\text{ind}}{\sim} (1-\theta_i) U(0, 1) + \theta_i Beta(1, ...
5
votes
1answer
49 views

Why does sampling from the posterior predictive distribution $p(x_{new} \mid x_1, \ldots x_n)$ work without having to average out the integral?

In a Bayesian model, the posterior predictive distribution is usually written as: $$ p(x_{new} \mid x_1, \ldots x_n) = \int_{-\infty}^{\infty} p(x_{new}\mid \mu) \ p(\mu \mid x_1, \ldots x_n)d\mu $$ ...
1
vote
0answers
27 views

Estimation of arithmetic Brownian motion volatility with transformed data

I want to estimate the volatility $\sigma$ of a process $(X_t)$ following an arithmetic Brownian motion, that is, for a constant time step $\Delta$, $X_{t+\Delta} = X_t + \sigma B_{\Delta}$ , where ...
2
votes
1answer
122 views

Gibbs sampling from a complex full conditional

I have a sampling question relating to Gibbs sampling of a complicated full conditional. Supposed I have a complicated full conditional that I want a single sample from $p(\theta_i$|$\theta_{-i}$, ...
1
vote
0answers
46 views

How does one determine which variables can be collapsed in Gibbs Sampling?

I am going through the derivation for Gibbs Sampling update equations for LDA. The claim is that $\theta_{d}$ (document specific topic distribution) and $\phi_k$ the topic-word distribution can be ...
0
votes
0answers
17 views

Interpretation of Gibbs sampling in Dirichlet Process posterior calculation

I wonder if anyone who is familiar with Gibbs sampling in the context of Dirichlet Process semi-parametric models could please help clear this question up. In Radford M. Neal's 2000 paper "Markov ...
2
votes
1answer
44 views

Efficiently sampling from mixture distribution posterior

I have the following model: $$ \begin{align} \pi_1\sim & \text{Unif}(0,1)\\ \lambda_1,\lambda_2\sim & \text{Ga}(1,1)\\ z_i\sim & \pi_1^{1(z_i=1)}\pi_2^{1(z_i=2)}\\ ...
1
vote
1answer
13 views

Is it possible for iterations to spike in Gibbs sampling?

After performing Gibbs sampling, I looked at a trace plot for one of my parameters and it appeared to spike at certain points. Is this possible or is it likely that I just coded my sampler wrong?
0
votes
1answer
23 views

Truncated prior leads to non-intuitive posterior

I am setting up a linear regression model for continuous data that is Normally distributed. For this model, I want to assume that my $\beta$ predictor is truncated to be positive, that is $$\beta \sim ...
0
votes
0answers
27 views

Is it possible for Metropolis sampling to converge to the wrong value?

I have simulated data under three parameters of interest, say a, b, c. The prior I put on c was a Gamma, so it only takes positive values. The full conditionals of a and b are known distributions, but ...
0
votes
1answer
45 views

Metropolis sampling (symmetric proposal distribution)

Can Metropolis sampling be used in conjunction with Gibbs sampling? So for example, if I have three parameters of interest, but only two of them have full conditionals that are known distributions, ...
0
votes
0answers
12 views

Gibbs within Collapsed Gibbs?

I have a model with variables $X_{1}, X_{2}, X_{3}, X_{4}$. I would like to sample it within a larger MCMC chain using: $(X_{1}, X_{2}) \sim P(X_{1}, X_{2})$ $(X_{3}, X_{4}) \sim P(X_{3}, X_{4} \mid ...
2
votes
2answers
188 views

Markov Chain Monte Carlo (MCMC) with transformed data

I want to obtain an estimate of a parameter $\Theta$ in a model for a random variable $X$ dependent on $\Theta$ with known but complicated likelihood $L(\Theta|X) = p(X|\Theta)$. $X$ is not directly ...
0
votes
0answers
27 views

plot log likelihood function evolution in mcmc simulations

Is it possible to plot log likelihood function evolution in mcmc simulations? I have a mixture model and its parameters are estimated using the gibbs sampling method in r environment and using the ...
2
votes
1answer
165 views

Gibbs sampling with mixed prior using a Metropolis-Hastings step

My questions are about a sampling procedure for fitting a Bayesian hierarchical model where one of the priors is a mixture distribution of discrete and continuous parts. The model is not my own but I ...
1
vote
2answers
98 views

Gibbs sampler implementation

I am just getting started with the Gibbs Sampler and came across an implementation from here and here and here. All of theses implementations are based on the first article. There is an inner loop ...
0
votes
1answer
28 views

Gibbs sampling: ancillary and sufficient parametrization

After asking a question about Gibbs sampling earlier, I have another one for you. I have not been able to find laymen's background on this, the only referenced use I've found for this is in ...
3
votes
1answer
167 views

Gibbs sampling convergence

In an astronomical context, the authors of a paper desire to use a Gibbs algorithm. Please note: I am inexperience in MCMC algorithms, and specifically in Gibbs sampling. What we want, in essence, is ...
0
votes
0answers
25 views

Gibbs Sampling Parallel convergence

I have a Gibbs Sampler. I am running 1000 parallel walks of the sampler. For any given walk I use the 2 value Kolmogorov Smirnov test to determine if convergence is reached within that instance. I ...
1
vote
0answers
42 views

Why is the posterior the stationary distribution of a Gibbs chain?

I'm having trouble understanding the setup here. I'm following Probabilistic Graphical Models by Koller and Friedman. They say that we wish to generate samples from the posterior distribution ...
1
vote
0answers
83 views

Efficiency in Metropolis Vs Gibbs sampling

I have read that Gibbs sampling is more efficient than Metropolis algorithm. Why? Is this due only to the fact the in Gibbs sampling the acceptance rate is $1$, so that the chain needs fewer ...
1
vote
0answers
35 views

What does it means of Normalization term of Gibbs distribution?

I am studying about Gibbs distribution concept and I am confusing a one term in that concept that is normalization term. According to the Hammersley–Clifford theorem, an random $x$ can equivalently be ...