The Gibbs sampler is a simple form of Markov Chain Monte Carlo simulation, widely used in Bayesian statistics, based on sampling from full conditional distributions for each variable.

learn more… | top users | synonyms

2
votes
0answers
23 views

When 2% of the Bayesian Model have not converged?

I have model with 20000 latent parameters, set up in a Gibb's sampler. 98% of the parameters and sometimes 99.5% of the parameters satisfy the Geweke convergence statistic, have low autocorrelation ...
3
votes
1answer
50 views

Gibbs sampling, what to use?

My question concerns Gibbs sampling. Suppose that I have three unknown quantities, $\mu, \sigma^2$ and $c$. I have given prior information and I have given the likelihood which allows me to compute ...
2
votes
1answer
30 views

Gibbs sampling and Conjugate Priors

Are conjugate priors required when performing Gibbs sampling?
0
votes
0answers
16 views

Gibbs Sampling for LDA example

Can someone provide an example of 1 (or more) iteration(s) of Gibbs sampling for LDA using real values? I have been searching for a while and I can't seem to find any good examples. Thank you.
4
votes
1answer
57 views

Gibbs sampling and conditional distribution

I need to simulate the posterior distribution of intraclass correlation coefficient $\pi(\rho|y)$ where $y$ is the data set and $\rho=\frac{\sigma_a^2}{\sigma_a^2+\sigma_e^2}$ with $\sigma^2_a\sim IG(\...
4
votes
0answers
225 views

Gibbs sampling deriving complete conditionals with mixture priors

My question is about the derivation of the complete conditionals for Gibbs sampling in a hierarchical model where some of the parameters are mixtures of point-masses and Normal distributions. The ...
0
votes
0answers
13 views

How can convergence (in distribution) be assessed in the context of multiple imputation by chained equations?

The MICE algorithm starts by randomly imputing the missing values in a dataset, and then proceeds to predict the missing values in each variable by modeling the relationship between the non-missing ...
1
vote
2answers
51 views

How does Gibbs sampling produce values for a variable using the univariate conditional probability?

I have a question about Gibbs sampling for generating samples. The Gibbs sampling algorithm is often stated. $x^0 = (x_1^0, x_2^0, \ldots, x_n^0)$ //initialize random values for $t=1$ in $T$ //...
0
votes
0answers
6 views

Forming distribution conditioned on many variables from single conditional marginals using copulas

I'm brainstorming about a data analysis project, part of which can be thought of as estimating a joint distribution from marginals, so I'd like to know whether I can use some copula techniques. ...
1
vote
1answer
49 views

Gibbs sampling in R

Let X a continous variable and Y a binary variable with joint distribution : $$p(x,y;\beta,\rho_1,\rho_2,\phi_1,\phi_2)=\frac{1}{Z(\beta,\rho_1,\rho_2,\phi_1,\phi_2)}\exp(-0.5 \beta x^2+1_{y=0}\rho_1 ...
0
votes
1answer
32 views

Gibbs sampling for mixed variables [duplicate]

Let X a continous variable and Y a binary variable with joint distribution : $$p(x,y;\beta,\rho_1,\rho_2,\phi_1,\phi_2)=\frac{1}{Z(\beta,\rho_1,\rho_2,\phi_1,\phi_2)}\exp(-0.5 \beta x^2+1_{y=0}\rho_1 ...
1
vote
2answers
63 views

Gibbs sampling and mixed distribution

For a project, I need to simulate from a joint distribution with both continuous and discrete variables that are dependent. The conditional distribution of any variable given the rest is known. I ...
0
votes
0answers
19 views

Conditional Distribution of Hidden Markov Model

I am trying to implement a Gibbs sampling algorithm for a toy Hidden Markov Model, but I am having trouble deriving the target conditional distribution. I am generating data through the following ...
0
votes
0answers
12 views

R: dlmGibbsDIG observation variance

I am working with the dlmGibbsDIG function (dlm library) and am having some problems with understanding the output of the formula. I analyzed a stock return (dY) and 4 independent variables (dX) with ...
0
votes
0answers
10 views

How to determine and use the sampling lag in the collapsed Gibbs sampler?

I am implementing a collapsed Gibbs sampler for LDA model. According to this technical note's word, average a number of samples, and often it is desirable to leave an interval of L iteration ...
0
votes
0answers
16 views

How does GibbsLDA++ ensure that we are sampling from a good posterior?

This is an extending of this question, which asked that whether we should do some estimating to ensure that we are really using a likely topic assignment instead of the one happened with low ...
1
vote
0answers
88 views

How to check the convergence in the collapsed Gibbs sampling of LDA? [closed]

I am trying to implement the LDA model fit by collapsed Gibbs sampling by myself. I have go through this article. And there is a clear pseudo code (section 5.5), ...
5
votes
0answers
31 views

Dirichlet process mixture MCMC

I'm reading Markov Chain Sampling Methods for Dirichlet Process Mixture Models by Radford M. Neal. Equation (3.6) states that If $c=c_{j}$ for some $j\neq i $: $P\left(c_{i}=c\;|\;c_{-i}, y_{i}, \...
3
votes
1answer
34 views

MCMC using GIBBS sampling: can different burn-in be used for different parameters?

I have run a stochastic volatility model with 4 parameters. I have used the Heidelberg and Welch convergence diagnostic. The result shows 3 out of 4 parameters have passed the stationary and half-...
3
votes
2answers
130 views

The Harris recurrence of a stepping-out slice-sampling-within-Gibbs MCMC

I want to use a multistage version of the MCMC here. That is, I want to use a Gibbs sampler to draw from a general joint distribution $p(x_1, x_2, x_3, \ldots)$ with a Gibbs step for each full ...
2
votes
0answers
57 views

Deriving mean and variance of the posterior distribution

I have a simple linear model: $y_{i}=\mu+e_{i}$ for $i=1,...,n$, where $P(e_{i})=w\mathcal{N}(0,\sigma^2) + (1-w)\mathcal{N}(0,k^2\sigma^2)$ with $w=0.9$, $k=10$ and $\sigma=0.1$. It can be understood ...
0
votes
0answers
28 views

Uniform sampling of constrained binary vectors by Gibbs sampling

General statement of the problem: Let $x,y$ be two binary vectors, connected by the following constrains: $$y=f(x),\qquad x=g(y)$$ That is, $x$ determines $y$, and $y$ determines $x$. There are many ...
0
votes
0answers
16 views

Gibbs Sampling for Multivariate Case

I understand the procedure for Gibbs sampling for bivariate case but I got confused about Gibbs sampler for multivariate normal distribution. I guess I should try to determine the conditional ...
0
votes
0answers
18 views

Derive Marginal Posterior to set up Gibbs-Sampler

I am currently trying to replicate a Hierarchical Model for multivariate returns proposed in the paper Portfolio selection using hierarchical Bayesian analysis and MCMC methods. However, in order to ...
2
votes
2answers
35 views

How to estimate biases from coin and dice using only observed dice throws in this setup?

To help me understand some concepts I'm learning in my first exposition to machine learning, I'm trying to tackle the following "simple" problem The setup of the problem is as follows: My friend ...
0
votes
0answers
19 views

Using two different methods for training and inferring

I have a model which is intractable to take derivatives wrt parameters and estimate them based on maximum likelihood. Even with an deterministic approximation like variational inference this is ...
0
votes
0answers
14 views

Independence yields exchange of summation and product in Gibbs Distribution

I am looking at the moment into statistical learning theory and encountered today the maximum entropy inference for k-mean clustering with a cost function given by: $$R(c)= \sum\limits_i || x_i - y_{...
3
votes
1answer
156 views

Sampling a random binary matrix with “Gaussian” probability distribution

Let $A_{ij}$ be a $n\times n$ random binary matrix with probability mass function $P(A)$ given by $$ \log P(A)=-\frac 12 \mathrm{tr}\left[\left(A-M\right)^TV\left(A-M\right)\right] + C, $$ where $M$ ...
2
votes
0answers
49 views

If all components of a hierarchical model have not converged, can we say that any parameters have truly converged?

I'm working with a hierarchical regression model of the following form similar to that presented in Peter D. Hoff's book, A First Course in Bayesian Statistical Methods: $\boldsymbol{Y}_j \sim \text{...
1
vote
0answers
39 views

Blocked Gibbs Sampling using Forward / Backward Algorithm

I am new to machine learning and have been reading about gibbs sampling. From my understanding, gibbs samples a single variable iteratively conditioned on all other variables. In blocked gibbs ...
0
votes
0answers
28 views

lasso gibbs sampler

Hi guys I tried to build a gibbs sampler for lasso regression. here is the model $y = Normal(x\beta,\sigma)\\ \beta = Laplace(alpha)\\ \sigma = Gamma(a,b)\\ \alpha = Gamma(c,d) $ I just used a ...
1
vote
1answer
40 views

Collapsed Gibbs Sampling in Mixture Models

I tried to learn how Gibbs sampling works on Mixture models by studying David Blei's notes: http://www.cs.columbia.edu/~blei/fogm/2015F/notes/mixtures-and-gibbs.pdf In the equation 28: $p(z_i = k| ...
4
votes
2answers
109 views

slice sampling within a Gibbs sampler

Questions My questions are: Is the following slice-sampling-within-Gibbs approach valid? Is there a good reference out there that uses, or better yet, justifies it? Context I'm trying to sample ...
2
votes
2answers
161 views

Gibbs sampler gets stuck in local mode

I am very new to statistics and trying to implement a Gibbs sampler. However, according to wikipedia https://en.wikipedia.org/wiki/Gibbs_sampling and this discussion thread http://metaoptimize.com/qa/...
1
vote
0answers
37 views

How to sample random variables (x,y) from a bivariate Cauchy distribution using a Gibbs sampler?

A bivariate Cauchy distribution is equivalent to a bivariate t-distribution with 1 degree of freedom.
7
votes
0answers
122 views

How to test if a cross-covariance matrix is non-zero

The background of my study: In a Gibbs sampling where we sample $X$ (the variable of interests) and $Y$ from $P(X|Y)$ and $P(Y|X)$ respectively, where $X$ and $Y$ are $k$-dimensional random vectors. ...
1
vote
1answer
26 views

Deriving Gibbs sampler for specific mixture model

Let $\theta_i$ be an indicator which is $0$ if score, $X_i$, is the same for both opponents, $1$ if different: $X_i|\theta_i \stackrel{\text{ind}}{\sim} (1-\theta_i) U(0, 1) + \theta_i Beta(1, \...
5
votes
1answer
56 views

Why does sampling from the posterior predictive distribution $p(x_{new} \mid x_1, \ldots x_n)$ work without having to average out the integral?

In a Bayesian model, the posterior predictive distribution is usually written as: $$ p(x_{new} \mid x_1, \ldots x_n) = \int_{-\infty}^{\infty} p(x_{new}\mid \mu) \ p(\mu \mid x_1, \ldots x_n)d\mu $$ ...
1
vote
0answers
28 views

Estimation of arithmetic Brownian motion volatility with transformed data

I want to estimate the volatility $\sigma$ of a process $(X_t)$ following an arithmetic Brownian motion, that is, for a constant time step $\Delta$, $X_{t+\Delta} = X_t + \sigma B_{\Delta}$ , where $...
2
votes
1answer
150 views

Gibbs sampling from a complex full conditional

I have a sampling question relating to Gibbs sampling of a complicated full conditional. Supposed I have a complicated full conditional that I want a single sample from $p(\theta_i$|$\theta_{-i}$, $...
1
vote
0answers
50 views

How does one determine which variables can be collapsed in Gibbs Sampling?

I am going through the derivation for Gibbs Sampling update equations for LDA. The claim is that $\theta_{d}$ (document specific topic distribution) and $\phi_k$ the topic-word distribution can be ...
0
votes
0answers
18 views

Interpretation of Gibbs sampling in Dirichlet Process posterior calculation

I wonder if anyone who is familiar with Gibbs sampling in the context of Dirichlet Process semi-parametric models could please help clear this question up. In Radford M. Neal's 2000 paper "Markov ...
2
votes
1answer
45 views

Efficiently sampling from mixture distribution posterior

I have the following model: $$ \begin{align} \pi_1\sim & \text{Unif}(0,1)\\ \lambda_1,\lambda_2\sim & \text{Ga}(1,1)\\ z_i\sim & \pi_1^{1(z_i=1)}\pi_2^{1(z_i=2)}\\ p(y_i|\lambda_1,\...
1
vote
1answer
14 views

Is it possible for iterations to spike in Gibbs sampling?

After performing Gibbs sampling, I looked at a trace plot for one of my parameters and it appeared to spike at certain points. Is this possible or is it likely that I just coded my sampler wrong?
0
votes
1answer
25 views

Truncated prior leads to non-intuitive posterior

I am setting up a linear regression model for continuous data that is Normally distributed. For this model, I want to assume that my $\beta$ predictor is truncated to be positive, that is $$\beta \sim ...
0
votes
0answers
31 views

Is it possible for Metropolis sampling to converge to the wrong value?

I have simulated data under three parameters of interest, say a, b, c. The prior I put on c was a Gamma, so it only takes positive values. The full conditionals of a and b are known distributions, but ...
0
votes
1answer
49 views

Metropolis sampling (symmetric proposal distribution)

Can Metropolis sampling be used in conjunction with Gibbs sampling? So for example, if I have three parameters of interest, but only two of them have full conditionals that are known distributions, ...
1
vote
0answers
16 views

Gibbs within Collapsed Gibbs?

I have a model with variables $X_{1}, X_{2}, X_{3}, X_{4}$. I would like to sample it within a larger MCMC chain using: $(X_{1}, X_{2}) \sim P(X_{1}, X_{2})$ $(X_{3}, X_{4}) \sim P(X_{3}, X_{4} \mid ...
2
votes
2answers
196 views

Markov Chain Monte Carlo (MCMC) with transformed data

I want to obtain an estimate of a parameter $\Theta$ in a model for a random variable $X$ dependent on $\Theta$ with known but complicated likelihood $L(\Theta|X) = p(X|\Theta)$. $X$ is not directly ...
0
votes
0answers
33 views

plot log likelihood function evolution in mcmc simulations

Is it possible to plot log likelihood function evolution in mcmc simulations? I have a mixture model and its parameters are estimated using the gibbs sampling method in r environment and using the ...