Questions tagged [gibbs]

The Gibbs sampler is a simple form of Markov Chain Monte Carlo simulation, widely used in Bayesian statistics, based on sampling from full conditional distributions for each variable or group of variables. The name comes from the method being first used on Gibbs random fields modeling of images by Geman and Geman (1984).

Filter by
Sorted by
Tagged with
2
votes
1answer
1k views

When should we use Gibbs Sampling in a deep belief network? Before or after fine-tuning?

Gibbs sampling allows for sampling a vector with a deep belief network. There are two steps to training a DBN for a supervised learning task: greedy unsupervised pre-training and supervised fine-...
2
votes
1answer
1k views

How can I estimate the precision of a normal using a Gibbs sampler?

I am trying to estimate the precision $\tau$ of a normal distribution with either WinBUGS or OpenBUGS: $c \sim \text{normal}(\mu,\tau)$ $\mu \rightarrow \lambda \cdot t^{-\beta}$ $\tau \sim \text{...
2
votes
1answer
657 views

Sampling from a Deep Belief Network: Treatment of biases in directed part of the model

When generating samples from a DBN, how do you handle the biases that have been learned for the layers below? I know that you normally perform a number of Block Gibbs sampling steps in the undirected ...
2
votes
1answer
437 views

Derivation of the posterior over topics in LDA

When studying the latent Dirichlet allocation, I am not very clear about some procedures in their deriving equations. Please refer to the attached figure, how to understand those two steps, marked as ...
2
votes
1answer
460 views

Non-conjugate improper uniform priors in Bayesian data analysis: how to handle infinite sums?

I've been working on Griddy Gibbs sampler (paper: Ritter and Tanner) and I've implemented it in R. But I've faced a problem when I started thinking on its uses in other contexts. If I try to use an ...
2
votes
1answer
95 views

Gibbs sampler for a multilevel model with no predictors in R

I'm working on multilevel models and want to know how they are estimated in R. For that purpose I'm reading amongst other things "Data Analysis Using Regression and Multilevel/Hierarchical Models" by ...
2
votes
1answer
43 views

Gibbs sampling allocations for time dependent observations from this model

I observe $N$ observations $\{x_{1,t_1}, \dots, x_{N,t_N}\}$ from a $k$ component Gaussian Mixture model. The $i$th observation is seen at time stamp $t_i$ and is distributed such that each $x_{i,t_i}|...
2
votes
1answer
87 views

Conditional Distribution to sample

Suppose I have six data points (n,x): (14,5), (13,4), (7,3), (10,5), (12,7), (20,13) which are realizations of binomial experiments on n trials with x successes respectively.. And I assume I ...
2
votes
1answer
314 views

Transition probabilities for Gibbs Sampling in a Markov Random Field

I am currently reading this paper on Restricted Boltzmann Machines. On page 22, Given a Markov Random Field $\mathbf{X} = (X_1,\ldots,X_N)$ w.r.t a graph $G = (V,E)$ where $V = \{1 \ldots N\}$ and $...
2
votes
1answer
27 views

Gibbs Sampler while fixing one parameter

I came across a problem for Gibbs sampler. Suppose I want to draw samples from $f(x,y,z)$, Can I use the following scheme to draw samples? Step 1, draw $f(x^{2t-1},y^t|z^{t-1})$. Step 2, draw $ f(...
2
votes
1answer
404 views

Posterior distributions of parameters in a AR(1) model

Consider a AR(1) model with states given by $x_t=\phi x_{t-1}+a_{t}$, $a_{t}\sim\mathcal{N}(0,\tau^2)$ and the observations given by $y_t=x_{t}+e_{t}$, $e_{t}\sim\mathcal{N}(0,\sigma^2)$ for $t=1,...
2
votes
1answer
375 views

Gibbs sampling with constraints

I am reading tutorials on Gibbs sampling for partition sampling in Dirichlet Process (Chinese Restaurant Process), and have been struggling to understand the terminology used in the tutorials. To ...
2
votes
2answers
533 views

Can one use the RTS (Rauch–Tung–Striebel) smoother to simulate latent factors (as opposed to Carter and Kohn procedure)?

It is a little surprising that I have not found anything online in the literature which clarified this. I am working on an MCMC Gibbs sampling procedure to calibrate a "dynamic factor model". One of ...
2
votes
1answer
451 views

Full conditionals - Gibbs Sampler

i want to draw samples from a 5-dimensional posterior distribution $f(k,\theta,\lambda,b_1,b_2|Y=y)$. From Bayes-Theorem there is the following relationship between posterior and likelihood: ...
2
votes
1answer
294 views

Gibbs sampling version for estimating the Dynamic Topic Model (DTM)?

The paper of Blei et Lafferty published at ICML'06 implements a (quite complicated) variational inference (VI) technique for estimating the parameters of the Dynamic Topic Model, see: http://pdf....
2
votes
1answer
631 views

Gibbs Sampling Detecting Change point in time series

I was reading through this one page paper on using Gibbs sampling for detecting a change point in a time series like data. While I understand the part where the $\lambda$ and $\phi$ are chosen from a ...
2
votes
1answer
491 views

Gibbs sampler for conditionals that are exponential: Example from Casella & George paper

I am trying to work out Example 2 from Casella and George's paper "Explaining the Gibbs Sampler" in R. The example is: ...
2
votes
1answer
740 views

Comparison of Slice sampling and Gibbs sampling

To me, the two are similar in the sense that slice sampling is just Gibbs sampling for the uniform distribution over the area under the plot of the density function. Is that right? I was wondering if ...
2
votes
0answers
27 views

Gibbs sampling for mixture with Dirichlet prior?

I want to sample from the distribution of a mixture distribution. The hierarchical model is $x_i\sim f$, where: $$f(x\mid \theta_1,\dots,\theta_p, w_1,\dots,\omega_p) = \sum_{j=1}w_p\varphi(x\mid\...
2
votes
0answers
35 views

FFBS algorithm for estimating mean log-return parameter in stochastic volatility jump model

I am currently attempting to replicate this model: https://arxiv.org/pdf/1809.01501.pdf in r. My (first) problem is regarding how to sample from conditional posterior for mu, $(μ_{(j)}|Y_n, J_{(j−1)}...
2
votes
0answers
311 views

An efficient way to generate multivariate normal distribution by Gibbs sampler? [closed]

When learning Gibbs sampler, the most used example is bivariate normal. But what if we want to simulate multivariate normal distribution? The computation (mean and variance) of conditional ...
2
votes
0answers
59 views

How to implement a M-H step in a Gibbs sampling

I am having trouble implementing a Metropolis Hastings step in a Gibbs sampling problem. The following code was taken from https://www.stat.colostate.edu/computationalstatistics/ Details: It is a ...
2
votes
1answer
47 views

Sampling from partial posterior

I'm reading a paper where the authors have something like the following as one step in their MCMC: $$ \begin{align} y&=\rho_1 x_1+\rho_2 x_2+\epsilon\\ z&=\beta\tilde{y}+u \end{align} $$ ...
2
votes
0answers
95 views

Can Bayesian Optimization solve this problem?

Suppose ${\bf{x}} = (x_1,\ldots,x_n)$ and $f({\bf{x}})\propto 1_A({\bf{x}}) \prod_{i=1}^n {x_i}^{\alpha_i-1} e^{-\beta_i x_i}$ , i.e. $f$ is proportional to the product of independent gamma ...
2
votes
0answers
259 views

Bayesian prior and posterior computation for a truncated normal

I have to deal with data in a Bayesian framework, ultimately devising a Gibbs sampler for inferring all my distributions parameters. Specifically, suppose I observe some univariate data distributed ...
2
votes
0answers
218 views

Missing data in Gibbs sampling for dynamic linear models

Suppose I have the following DLM: $x_t = \Phi x_{t-1} + w_t$ $y_t = A x_t + v_t$ $x_0 \sim N(\mu_0,\Sigma_0)$ $w_t \sim N(0,Q)$ $v_t \sim N(0,R)$ Let $\Theta = \{\mu_0,\Sigma_0,\Phi,Q,A,R\}$. I ...
2
votes
0answers
348 views

Gibbs sampling in the Hierarchical Dirichlet Process

For an inference problem using a Dirichlet Process prior, one can derive a "basic" Gibbs sampling scheme, where we have a conditional for any parameter $\theta_i$ given the samples $x_i$ and all the ...
2
votes
0answers
26 views

A verifiable and teachable Gibbs example

I am attempting to construct a teachable example of Gibbs sampling that I can also relate to how it might be used on an actual dataset and yet could also be verified analytically by students with ...
2
votes
0answers
133 views

Seeking help in Bayesian Mixed Effects Model

I am implementing a Bayesian Mixed Effects model in my research problem. The model is written as, $y_i = X_i(\alpha + \beta_i) + \epsilon_i$, where $i = 1, 2, \ldots, m$ is the index of response, $j = ...
2
votes
0answers
1k views

Deriving mean and variance of the posterior distribution

I have a simple linear model: $y_{i}=\mu+e_{i}$ for $i=1,...,n$, where $P(e_{i})=w\mathcal{N}(0,\sigma^2) + (1-w)\mathcal{N}(0,k^2\sigma^2)$ with $w=0.9$, $k=10$ and $\sigma=0.1$. It can be understood ...
2
votes
0answers
106 views

If all components of a hierarchical model have not converged, can we say that any parameters have truly converged?

I'm working with a hierarchical regression model of the following form similar to that presented in Peter D. Hoff's book, A First Course in Bayesian Statistical Methods: $\boldsymbol{Y}_j \sim \text{...
2
votes
0answers
116 views

Estimation of arithmetic Brownian motion volatility with transformed data

I want to estimate the volatility $\sigma$ of a process $(X_t)$ following an arithmetic Brownian motion, that is, for a constant time step $\Delta$, $X_{t+\Delta} = X_t + \sigma B_{\Delta}$ , where $...
2
votes
0answers
402 views

Why is the posterior the stationary distribution of a Gibbs chain?

I'm having trouble understanding the setup here. I'm following Probabilistic Graphical Models by Koller and Friedman. They say that we wish to generate samples from the posterior distribution $P(\...
2
votes
0answers
92 views

Can I use Adaptive MCMC in any setting?

In time series econometrics and finance, most Bayesian authors approximate their models with a Gibbs Sampler, this is especial true for state space models, SV and so forth. The dimensionality of ...
2
votes
0answers
244 views

Gibbs sampling version for estimating Hierarchical Double Dirichlet Process Mixture of Gaussian Processes

I'm trying to implement Gibbs sampling to estimate the parameters of the following non-parametric model: $$\begin{align*} \beta|\gamma & \sim \text{GEM}(\gamma)\\ k_t|\beta & \sim \beta\\ \pi|\...
2
votes
0answers
197 views

Conjugate prior for multivariate with known mean and covariance known to a constant

I have a linear trend model (evolving mean and slope) embedded in a larger state space time series model that I would like to constrain to be a spline. With that assumption, the mean and trend ...
2
votes
0answers
139 views

Understanding a measure of convergence of MCMC simulations

I am trying to better understand better the Gelman/Rubin measure of convergence of MCMCs. The method starts off by defining two quantities: $B$ and $W$. $B$ is said to be the between chain variance (...
2
votes
0answers
172 views

Gibbs sample from AR(1) of exogenous input

I am trying to fit a model where there is a sequence of exogenous "shocks", $X_1, X_2, ..., X_T$, and a AR(1) of these shocks explain $Y_1, Y_2, ..., Y_T$. Specifically, Data (known): $X_1, X_2, ...,...
2
votes
0answers
91 views

Mixture of probits: understanding truncated-based likelihoods

I am trying to implement a mixture model of probits to infer the best decision boundary for every latent subpopulation. When doing Gibbs sampling, we eventually have to compute $P(y^* | w_c)$ where $...
2
votes
0answers
413 views

Practical Implementation of Gibbs Sampling in Latent Dirichlet Allocation

In the collapsed Gibbs sampling version of LDA, the posterior distribution of topic assignments for each word is sampled. From what I have read (e.g. http://people.cs.umass.edu/~wallach/courses/s11/...
2
votes
1answer
83 views

What will happen if we sample the most probable value in the Gibbs sampling?

I am now working with the Gibbs sampling. One problem that puzzled me is that when we use the Gibbs sampling, we always sample randomly from the conditional probability. What will happen if we sample ...
2
votes
0answers
271 views

Sampling Stationary Vector Autoregression coefficients while Gibbs Sampling

I have been estimating a Bayesian Vector Autoregression using Gibbs Sampling. When constructing the posterior predictive distribution, I have noticed that when the simulated coefficients from the MCMC ...
2
votes
0answers
1k views

Gibbs sampler on the precision (with a gamma prior) in a hierarchical Bayesian model doesn't converge

I am deriving a Gibbs sampler with a model similar to the model in this paper (a graphical model is shown in page 4). To put it simple, my question only concerns $w_i$ (a $K$-dimensional vector drawn ...
2
votes
0answers
236 views

Mathematical reference for the convergence in distribution of the Gibbs sampler

This question is in some sense the intersection of this question and this question. I have read up on the Gibbs sampler, and am now asking for an introduction to the Gibbs sampler for mathematicians. ...
1
vote
2answers
150 views

Gibbs algorithm using negative binomial produces NAs

I have the following full conditionals distributions: $$ X_2|X_1=x_1\sim Bin(x_1,p)\\ X_1|X_2=x_2\sim NegBin(x_2,p) $$ So I'm using the following code to generate a sample from each one: ...
1
vote
1answer
220 views

Gibbs sampling for reducible chain

I am new to Gibbs sampling and I ran into a problem with irreducibility. For the Gibbs sampler to work the Markov chain has to be irreducible. But that assumption is not satisfied in my probability ...
1
vote
1answer
357 views

When can the collapsed Gibbs sampler be implemented?

I understand Gibbs sampling is a means of statistics inference, and it seems that sometimes certain variables can be integrated out in the sampling process, known as collapsed Gibbs sampling. I really ...
1
vote
1answer
266 views

Expected value of an estimate done using Gibbs sampling

This is related to this question. I am concerned about the expected value of a function estimated using sampling. If the samples were obtained by a sampling method such as Gibbs sampling, and if I ...
1
vote
1answer
199 views

MCMC sampling with sum constraints

I'm interested in sampling a collection of variables with a sum constraint on them. For a simplified example: Prior: $X \sim \mathcal{N}(0, 1)$ $Y \sim \mathcal{N}(0, 1)$ Observation: $X + Y = 1$ ...
1
vote
2answers
902 views

Implementing Latent Dirichlet Allocation - notation confusion

I am trying to implement LDA using the collapsed Gibbs sampler from http://www.uoguelph.ca/~wdarling/research/papers/TM.pdf the main algorithm is shown below I'm a bit confused about the notation ...