Questions tagged [gibbs]

The Gibbs sampler is a simple form of Markov Chain Monte Carlo simulation, widely used in Bayesian statistics, based on sampling from full conditional distributions for each variable or group of variables. The name comes from the method being first used on Gibbs random fields modeling of images by Geman and Geman (1984).

Filter by
Sorted by
Tagged with
0
votes
1answer
99 views

Sampling from conditional posterior - continuous and discrete terms

I'm working through Hierarchically Supervised LDA by Perotte et all (2011). The conditional posterior I'm supposed to sample values $z_i$ from, however, is zero almost everywhere. To see why, lets ...
41
votes
4answers
6k views

OpenBugs vs. JAGS

I am about to try out a BUGS style environment for estimating Bayesian models. Are there any important advantages to consider in choosing between OpenBugs or JAGS? Is one likely to replace the other ...
3
votes
1answer
1k views

Gibbs measure and normal distribution

On Wikipedia, the Gibbs measure defines the probability as: $$ P(X=x) = \frac{1}{Z(\beta)}\exp(-\beta E(x)) $$ Now, the familiar form of the normal distribution is: $$ P(x) = \frac{1}{\sqrt{2\pi}\...
1
vote
1answer
202 views

Sampling from a posterior with Gibbs sampling

In an image processing class, I dont really get behind the idea how to 'sample from a posterior' with Gibbs sampling. We have a posterior distribution: $f(z_1, .. ,z_n \mid x_1,.. ,x_n) := f(z \mid x)...
3
votes
1answer
590 views

Gibbs sampling an Ising model with 0s and 1s

One of my problems in one of my courses ask to sample a 20 dimensional vector of 0s and 1s, $\{0,1\}^{20},$ when they are distributed as $$ \pi(x) = \exp\left\{-\beta \sum_{i=1}^{19} |x_{i+1}-x_i| \...
1
vote
0answers
29 views

Gibbs sampling with expectations instead of sampling

I see there is something called Iterated Conditional Models (ICM), which is a sort of Gibbs sampling where, instead of sampling, we use the value that maximizes the conditional. That is: ...
3
votes
1answer
483 views

Blocked Gibbs Sampling using Forward / Backward Algorithm

I am new to machine learning and have been reading about gibbs sampling. From my understanding, a Gibbs algorithm samples a single variable iteratively conditioned on all other variables. In blocked ...
0
votes
1answer
53 views

Priors on Taylor Expansion series

I'm wondering what priors can i choose for a Taylor series as follows: $\theta_{1}+\theta_{2} (y-\alpha) + \theta_{3} (y-\alpha)^2$ What priors should I use for updating these parameters ($\theta_{1},...
1
vote
0answers
227 views

Collapsed Gibbs sampler on Hierarchical Dirichlet Process Mixture Model

I am trying to design a collapsed Gibbs sampler on a mixture model based on Hierarchical Dirichlet Process ($g\sim DP(\gamma, b)$, $\pi\sim DP(\alpha, g)$ ). Should I resample from the posterior of ...
0
votes
1answer
114 views

Rewrite conditional formula with three variables using Bayes formula

In equation (5) on page 3 on this paper a conditional probability is rewritten using Bayes' formula. I started using this answer Can I rewrite conditional probability of three variables like this? ...
4
votes
1answer
73 views

Sampling from multivariate normal conditional on a negative minimum

Let $X\sim \mathcal{N}(\mu,\Sigma)$, where $\mu\in\mathbb{R}^n$ and $\Sigma\in\mathbb{R}^{n\times n}$. How can I efficiently sample from $X | {\min{X}\le 0}$? (I.e. from the distribution of $X$ ...
0
votes
0answers
24 views

Gibbs sampler from from $p(x) = C g(x)$ with $C$ unknown and discrete elements in $X$

By using the Hasting-Metropolis method, is there a way to draw samples from a distribution of this form: $$p(\textbf{x}) = C g(\textbf{x})$$ For $x$ being two dimensional and discrete. The reason that ...
8
votes
1answer
722 views

Can I subsample a large dataset at every MCMC iteration?

Problem: I want to perform a Gibbs sampling to infer some posterior over a large dataset. Unfortunatelly, my model is not very simple and thus sampling is too slow. I would consider variational or ...
1
vote
1answer
52 views

Gibbs sampler -transformation of conditional posterior

If my conditional posterior $\pi(\sigma^{-2}|\mathbf{y },\mu)\sim Gamma(a,b)$, how can I get the conditional posterior $\pi(\sigma^{2}|\mathbf{y },\mu)$ with a transformation? The reason I ask is ...
2
votes
1answer
208 views

Gibbs sampling from conditional full posterior distribution

I am reading the paper by Willemsen et al (2015), "A multivariate Bayesian model for embryonic growth", Statistics in Medicine, 34:8, 1351–1365 where they define the posterior distribution as, \begin{...
6
votes
2answers
5k views

Gibbs sampling how to sample from the conditional probability? Bayesian model

I want to learn Gibbs sampling for a Bayesian model. How can I sample the variable from the conditional distribution? In this example, arrow means dependent; for example, ...
0
votes
1answer
159 views

use inverse Wishart for variance in MCMC

When you have a posterior that looks like as one step in Gibber Sampler $P(\xi | \Sigma_\xi, \theta) ∝ exp\{-1/2 \xi\Sigma_\xi^{-1}\xi\}P(data | \xi, \theta)$ Do you always assume inverse Wishart ...
1
vote
1answer
364 views

which R package is good for gibbs sampler when the likelihood function is complex

I see a lot of examples using MCMC to solve for posterior distribution when the likelihood is simply one of linear regression. What if the likelihood is an ugly, complex function. Which R package can ...
5
votes
1answer
97 views

Estimation of Bayesian Models

I'm trying to get into Bayesian model estimation (I'm interested in posterior parameter distributions). I could get away with Metropolis-Hastings and Gibbs Sampling for models with few parameters (<...
2
votes
0answers
322 views

Gibbs sampling in the Hierarchical Dirichlet Process

For an inference problem using a Dirichlet Process prior, one can derive a "basic" Gibbs sampling scheme, where we have a conditional for any parameter $\theta_i$ given the samples $x_i$ and all the ...
1
vote
0answers
112 views

Gibbs Sampling / Monte Carlo with Weights

Consider pairs of data and their population weights $(y_i, \omega_i), i = 1, 2, \dots$ alongside some hierarchical structure, $$y_i\leftarrow\theta_{i(j)} \leftarrow \gamma$$ where $i(j)$ is perhaps ...
1
vote
0answers
178 views

Get different results with different sampling order in Gibbs sampling: what could be wrong?

In sampling a complex spatio-temporal model by Gibbs sampling, I found if I change the order of sampling (for example, to sample $P(\theta_1,\theta_2|D)$, in one try, I sample $\theta_1\sim P(\theta_1|...
1
vote
2answers
1k views

Unsupervised Bayesian naive Bayes

I'm reading a paper Gibbs sampling for the uninitiated. In this paper, the authors try to use Gibbs sampling for a bayesian naive bayes model. They formalize the model as a graphical model in page 8. ...
0
votes
1answer
141 views

Eq. 45-46 in Gibbs Sampling for the Uninitiated

I am trying to figure out how Eq. 45 simplifies to Eq. 46 in the paper - "Gibbs Sampling for the Uninitiated" by Resnik and Hardisty.www.cs.umd.edu/~hardisty/papers/gsfu.pdf (page 15) Eq. 45 $$ \...
2
votes
1answer
147 views

Is Coordinate Ascent algorithm related to Gibbs Sampling in some way?

I wonder if only me feel there are certain connections between them, I googled it for a long time, but found no where mentioned these two method. But to me, they indeed looked so related, Could anyone ...
2
votes
2answers
334 views

Gibbs Sampler, Bivariate Normal, Subchain

In Example 7.1 of "Introducing Monte Carlo Methods with R", the authors write $(X,Y)\sim N\Bigg((0,0),\begin{pmatrix}1 &\rho \\ \rho & 1\end{pmatrix}\Bigg)$ Then, Given $x_t$, $Y_{t+1}\mid ...
0
votes
2answers
254 views

stationarity of vector autoregression and Gibbs sampling

I'm estimating a vector autoregression (VAR) using Gibbs sampling. At each iteration, I'd like to check the coefficients to ensure the VAR is stationary. An older, related question has been posted ...
1
vote
1answer
88 views

Gibbs Sampler Consistency

Simple question that I haven't found easily online: why are the estimates obtained from a Gibbs sampler consistent (converge to the true probabilities)?
1
vote
0answers
27 views

Sampling from a truncated random effects distribution

How would one sample observations $T_{ij} = U_i + \varepsilon_{ij}$, where the distribution of $U_i, \varepsilon_{ij}$ are known and mutually independent, condition on the fact that $L_{ij} \le T_{ij} ...
2
votes
1answer
27 views

Gibbs Sampler while fixing one parameter

I came across a problem for Gibbs sampler. Suppose I want to draw samples from $f(x,y,z)$, Can I use the following scheme to draw samples? Step 1, draw $f(x^{2t-1},y^t|z^{t-1})$. Step 2, draw $ f(...
2
votes
2answers
2k views

Metropolis-Hastings Algorithm within Gibbs Sampling

I have this $f$ function below. $$ f(x_1,x_2)\propto \left(\dfrac{x_1}{x_2}\right)\left(\dfrac{\alpha}{x_2}\right)^{x_1-1}exp\left\{-\left(\dfrac{\alpha}{x_2}\right)^{x_1} \right\}I_{R^+}(x) $$ where ...
3
votes
1answer
185 views

What is a hierarchical model that can estimated via the Metropolis-Hastings Algorithm but not the Gibbs Sampler?

My understanding of the differences between MH and Gibbs Samplers is that a Gibbs Sampler is usually used when the full conditionals are present to us. In other words, it is a known distribution, so ...
3
votes
0answers
492 views

Bayesian estimation using Gibbs sampling for financial models

I am trying to do Gibbs sampling, from this paper. This is a CIR financial model, I want to do Gibbs on its parameters: $$y(t+{\Delta}^{+})=y(t)+(\alpha-\beta y(t)){\Delta}^{+}+\sigma \sqrt{y(t)}{\...
1
vote
1answer
378 views

Sampling from the joint distribution p(x,y) when y = f(x)

Suppose I want to sample from the joint distribution $p(X, Y)$, where $X$ is a random variable and $Y = f(X)$ where $f$ is a known function of $X$. However, sampling from $p(X,Y)$ directly is hard. ...
2
votes
1answer
374 views

Gibbs sampling on the use of gamma distribution

I am trying to reproduce a Gibbs sampling but some points are unclear to me. Let's assume a linear model of the following form $$y=x\beta + \epsilon$$ The prior distribution for the beta ...
2
votes
2answers
2k views

How does Gibbs sampling produce values for a variable using the univariate conditional probability?

I have a question about Gibbs sampling for generating samples. The Gibbs sampling algorithm is often stated. $x^0 = (x_1^0, x_2^0, \ldots, x_n^0)$ //initialize random values for $t=1$ in $T$ //...
3
votes
0answers
710 views

Bayesian estimation of Dynamic Regression with AR(1) parameters

I would like to draw (Bayesian) inference in a dynamic linear regression with regression parameters following independent AR(1) processes $\beta_{t,i} = \mu_i+\beta_{t-1,i}+w_{t,i}$. However, I ...
1
vote
2answers
157 views

throwing away all Gibbs samples after approximation

This is more of a theory question, consider: $$P(w_1|D)=\int P(w_1|S)P(S|D)d(S)$$ which we approximate via Gibbs sampling $S$ (assume the initial state of the Gibbs sampler is denoted by $M_0$), ...
3
votes
1answer
167 views

Calculating conditional probability

Let's consider I have the pair of distributions: \begin{align} X|t &\sim Binomial(n,t) \\ t &\sim Beta(a,b)$ \end{align} Here $n,a,b$ are known. I need to construct conditional probability ...
2
votes
0answers
107 views

Estimation of arithmetic Brownian motion volatility with transformed data

I want to estimate the volatility $\sigma$ of a process $(X_t)$ following an arithmetic Brownian motion, that is, for a constant time step $\Delta$, $X_{t+\Delta} = X_t + \sigma B_{\Delta}$ , where $...
4
votes
1answer
198 views

handling metropolis hastings rejection during a Gibbs sweep

Suppose I have a MCMC involving a 2 step Gibbs sampler. The first part uses metropolis hastings to find the next parameter value. If during one sweep, the result for the first part is a rejections, ...
2
votes
0answers
26 views

A verifiable and teachable Gibbs example

I am attempting to construct a teachable example of Gibbs sampling that I can also relate to how it might be used on an actual dataset and yet could also be verified analytically by students with ...
1
vote
0answers
58 views

Methods to solve a compound distribution MLE I can't write in closed form

What are some methods of solving a MLE parameter estimate where the likelihood function can't be written in closed form? I have a compound distribution that I don't have a closed form for (no ...
1
vote
1answer
554 views

How to do MC integration from gibbs sampling of posterior

I'm a beginner of MCMC. Two questions confused me. If I know the posterior distribution, and from the Gibbs sampling, I got the sampled parameter, so How to draw the histogram with y axis as ...
1
vote
1answer
490 views

Generating Sample Path of HMM via Gibbs Sampling

I have a question regarding section 7.1.1 here. I have two questions to ask you. What does the following sentence mean? we shall be drawing values for $C_T,C_{T-1},\cdots,C_1$ in order. How the ...
1
vote
0answers
49 views

Gibbs sampling confusion

I'm just wondering if i'm doing this process correctly i'm a little confused as to the answers i'm getting: I have $P(x=0,y=0) = P(x=1,y=1) = 0.5$ and $P(x=0,y=1) = P(x=1,y=0) = 0$ I calculated $P(...
4
votes
0answers
616 views

Gibbs sampling for spike and slab priors

In Spike and slab variable selection (equation 4) there is a model setup of the form $\beta_k | \lambda_k, \tau_k \sim \text{Normal} (0, \lambda_k \tau_k^2)$ $\lambda_k | \nu_0, w \sim (1-w)\delta_{\...
3
votes
0answers
292 views

When Gibbs Sampling is fast/slow to converge?

Are there any heuristic/theories showing that on what kind of Bayesian models the convergence of Gibbs sampling is fast/slow? For example, from my limited experience, I feel (may be wrong) when a ...
1
vote
1answer
190 views

Conditional posterior probability density (steps)

I'm trying to understand how to condition a probabilistic posterior distribution. Consider the following probability density: $$ p(\alpha, \beta | y) = \prod_{i=1}^n (\alpha+\beta t_i)^{y_i}e^{-(\...
3
votes
0answers
588 views

Criteria in determining “step size” of Metropolis-hasting algorithms

I am training a complex Bayesian model using Gibbs sampling and Metropolis-Hasting algorithm. Most of the parameters are directly sampled by using conjugate priors except for 3 params which are ...