The Stack Overflow podcast is back! Listen to an interview with our new CEO.

Questions tagged [gibbs]

The Gibbs sampler is a simple form of Markov Chain Monte Carlo simulation, widely used in Bayesian statistics, based on sampling from full conditional distributions for each variable or group of variables. The name comes from the method being first used on Gibbs random fields modeling of images by Geman and Geman (1984).

Filter by
Sorted by
Tagged with
0
votes
0answers
17 views

the convergence speed for a Markov chain

For a metropolis hastings algorithm, suppose that the stationary distribution is defined as the Gibbs Boltzmann distribution $\pi_T(x)= \frac{1}{Z_T}e^{-\frac{V(x)}{T} }$ where $Z_T = \sum_{y\in V} e^{...
1
vote
1answer
190 views

What does it means of Normalization term of Gibbs distribution?

I am studying about Gibbs distribution concept and I am confusing a one term in that concept that is normalization term. According to the Hammersley–Clifford theorem, an random $x$ can equivalently be ...
0
votes
1answer
23 views

Inference of the collapsed Gibbs sampling for LDA

I am trying to understand the inference procedure of collapsed Gibbs sampling for LDA model. I refer to this document and LDA wiki page. I cannot figure out how does it simplify the sample equation ...
0
votes
0answers
13 views

Dirichlet Process mixture model with independent features

I'm trying to construct a Dirichlet process mixture model for clustering where the samples have independent features. In other words, to evaluate the likelihood of sample $x_i$, I would compute $\...
0
votes
1answer
19 views

Gibb's sampling where target prob distribution is itself a conditional joint distribution - p(x,y|t)

I'm new to Gibb's sampling and need basic guidance. Say p1,p2,q-> are Gaussian variables. p1->q<-p2 and q->x where x is a discrete variable (either 1 or 0). How do I go about Sampling (using ...
1
vote
1answer
23 views

Assign an error to the parameters of MAP estimate

Through a MCMC Gibbs sampler I obtain $M$ chains of the parameters vector $\mathbf{\theta}$, meaning that each component of $\mathbf{\theta}$ is the value of one parameter at a given iteration. ...
0
votes
1answer
24 views

Gibbs sampling: Conjugate prior for a two component known-unknown variance?

If I have a normally distributed variable $\mathcal{N}(\mu,\frac{1}{\tau})$ with fixed $\mu$ then the conjugate prior for an unknown $\tau$ is then $\mathcal{Ga}(\frac{n}{2}+\alpha, \beta + \sum_i \...
0
votes
1answer
107 views

GIBBS sampling : do samples for the subset of variables approxiamate the related marginal distribution?

I'm reading the page Gibbs sampling on wikipedia. I really don't understand why the following statement is true. "The marginal distribution of any subset of variables can be approximated by simply ...
1
vote
0answers
20 views

Is there something wrong with my Bayesian hierarchical spatio-temporal model?

I built a Bayesian spatio-temporal model and one of the parameters to be estimated is the random spatial effects s. The random spatial effect is assigned an intrinsic conditional autoregressive prior (...
1
vote
1answer
542 views

Influence of word counts from DTM on LDA with Gibbs Sampling

I'm trying to wrap my head around Topic Modeling based on LDA with Gibbs sampling (Griffiths, Steyvers 2004: Finding Scientific Topics). What struck me when reading some Python implementations like ...
1
vote
1answer
141 views

How does this Sampler work for the Concentration parameter of Dirichlet Process?

I am puzzled by how this Gibbs sampler on section 6 of Escobar & West (1995) works. To put it in simple words, the aim is to sample $\alpha$. The defined terms are: $$\eta\sim \texttt{Beta}(a,b)$$ ...
4
votes
1answer
77 views

Is the joint distribution $P_{XY}(x,y)$ determined from the conditionals $P_{X|Y}(x|y)$ and $P_{Y|X}(y|x)$?

For simplicity assume that $X,Y$ are discrete, finite, random variables, with joint distribution $P_{XY}(x,y) = \mathbb{P}(X=x\wedge Y=y)$. Now suppose that we do not know $P_{XY}(x,y)$, but are ...
41
votes
4answers
7k views

OpenBugs vs. JAGS

I am about to try out a BUGS style environment for estimating Bayesian models. Are there any important advantages to consider in choosing between OpenBugs or JAGS? Is one likely to replace the other ...
0
votes
0answers
30 views

sampling from joint distribution to recover marginal distribution

I'm going through Bayesian Core and have gotten stuck at this remark on page 233: " A first remark that motivates the use of the Gibbs sampler is that, within structures such as $$ \pi(x_1) = \int \...
8
votes
1answer
76 views

What is the correct way to write the elastic net?

I am confused about the correct way to write the elastic net. After reading some research papers there seems to be three forms 1) $\exp\{-\lambda_1|\beta_k|-\lambda_2\beta_k^2\}$ 2) $\exp\{-\frac{(\...
0
votes
0answers
16 views

Any relationship between MAP full conditionals and MAP joint?

Context: Bayesian model that one can draw posterior samples from via Gibbs sampling of the relevant full conditionals. Question: Can anything be said (i.e. bounds, conditions for equality) about the ...
0
votes
0answers
12 views

Gibbs Sampling attempt at a simple Coxian distribution

I have the following Coxian model for inter-arival times ($x_i$) that has $C_x^2 < 1$: $$ p(x_i\mid \lambda,\theta) = \theta \lambda^r x_i^r e^{-\lambda x_i} + (1-\theta)\lambda e^{-\lambda x_i} $...
1
vote
0answers
33 views

Understand the Holmes and Held (2006) Bayesian probit MCMC algorithm

Holmes and Held (2006) suggest a simple approach to reduce autocorrelation in the MCMC algorithm proposed by Albert and Chib (1993). HH (2006) propose to update $\beta$ and $z$ jointly, making use of ...
0
votes
0answers
41 views

Gibbs Sampler for GMM

In Rasmussen's paper it is introduced a Gibbs sampler to make inference about a standard Gaussian Mixture Model. To simplify, assume the 1-d case with basic hierarchical structure, that is: $x_i|...
3
votes
1answer
669 views

Gibbs sampling and Conjugate Priors

Are conjugate priors required when performing Gibbs sampling?
2
votes
0answers
24 views

Gibbs sampling for mixture with Dirichlet prior?

I want to sample from the distribution of a mixture distribution. The hierarchical model is $x_i\sim f$, where: $$f(x\mid \theta_1,\dots,\theta_p, w_1,\dots,\omega_p) = \sum_{j=1}w_p\varphi(x\mid\...
0
votes
1answer
152 views

How to create a distribution and sample?

Suppose we are given some small set of data on bundles of electrical wires and increasing voltages run through them, and we note how many of the individual wires fail. So for example, a large data ...
0
votes
0answers
16 views

Shape of parameters marginal posterior in hierarchical Bayes model

Consider a generic hierarchical Bayes model with data $y_i\sim p(y|\theta_i)$, dependent of parameters $\theta_i\sim p(\theta|\phi)$ and hyperparameters $\phi\sim p(\phi)$. Furthermore, assume that $\...
0
votes
1answer
42 views

Clusters keep switching in Gibbs sampling of Dirichlet Process Mixture Model

All the code and data for this question is on GitHub (stackexchange.R script). I've got multivariate Bernoulli data that I'd like to analyse using Bayesian Mixture ...
3
votes
1answer
43 views

Auxiliary variable Gibbs sampler

Suppose we want to sample from a pdf $f(x_1,x_2)$. It's easy to sample from $x_1 \vert x_2$, but not $x_2 \vert x_1$, so we introduce an auxiliary variable $u$ such that $\int f(x_1,x_2,u) du = f(x_1,...
0
votes
1answer
218 views

Inferring GMM parameters with Gibbs Sampling

On my book, "Machine Learning A Probabilistic Approach". It's stated that is straightforward to derive a Gibbs sampling algorithm to fit a mixture model, especially if we use conjugate priors. So ...
1
vote
2answers
139 views

Gibbs algorithm using negative binomial produces NAs

I have the following full conditionals distributions: $$ X_2|X_1=x_1\sim Bin(x_1,p)\\ X_1|X_2=x_2\sim NegBin(x_2,p) $$ So I'm using the following code to generate a sample from each one: ...
0
votes
0answers
40 views

Metropolis-within-Gibbs for parametric inference of a regressive model

I have a regressive model of this form \begin{equation} y=f(\theta)+\varepsilon \end{equation} to describe observations $y$, with noise $\varepsilon$ and a parametric function $f$ with parameters $\...
0
votes
0answers
31 views

Can Gibbs Sampling be used to generate synthetic data from the posterior distribution?

I am working with a few datasets. I would like to expand some of them somehow by creating synthetic data. Is it possible to use a Gibbs Sampler to, by sampling a given distribution of the original ...
0
votes
1answer
25 views

Does Gibbs sampling require to know the partition function?

As the title suggest, does gibbs sampling require to know the partition function? For example, if I want to sample variable $a$ and I have worked out $p(a|rest) \propto f(a|rest)$ where $rest$ ...
0
votes
0answers
24 views

Conjugate prior for DPGMMs using Gibbs sampling

I am using Gibbs sampling to infer DPGMMs. The prior for multivariate Gaussians is Normal-inverse Wishart. But it turns out that the covariances are not estimated accurately. Here is codes and results....
0
votes
2answers
202 views

How is the accuracy of the results of Gibbs sampler measured?

How is the accuracy of the results of Gibbs sampler measured? Most resources merely say to iterate it $k$ times. But how does one infer the accuracy of the result?
2
votes
0answers
35 views

FFBS algorithm for estimating mean log-return parameter in stochastic volatility jump model

I am currently attempting to replicate this model: https://arxiv.org/pdf/1809.01501.pdf in r. My (first) problem is regarding how to sample from conditional posterior for mu, $(μ_{(j)}|Y_n, J_{(j−1)}...
2
votes
1answer
63 views

What's the role of the data in Gibbs Sampling?

Trying to wrap my mind around Gibbs Sampling. Across many answers in this same forum, I constantly notice that the examples shown do not actually require an observed data set (First example (with R ...
92
votes
3answers
55k views

Can someone explain Gibbs sampling in very simple words? [duplicate]

I'm doing some reading on topic modeling (with Latent Dirichlet Allocation) which makes use of Gibbs sampling. As a newbie in statistics―well, I know things like binomials, multinomials, priors, etc.―,...
0
votes
0answers
19 views

precision or variance of a Gamma distribution in a Gibbs Sampler?

I want to confirm my thinking on a quick question I have regarding the Normal-Gamma Gibbs sampler that we see so often, but I am unable to find a satisfactory answer. If we are interested in ...
2
votes
1answer
385 views

Posterior distributions of parameters in a AR(1) model

Consider a AR(1) model with states given by $x_t=\phi x_{t-1}+a_{t}$, $a_{t}\sim\mathcal{N}(0,\tau^2)$ and the observations given by $y_t=x_{t}+e_{t}$, $e_{t}\sim\mathcal{N}(0,\sigma^2)$ for $t=1,...
0
votes
0answers
24 views

Gibbs sampling for drawing samples and estimating parameters

I'm learning Bayesian inference by myself and having a difficulty for understanding Gibbs sampling. From what I understood, Gibbs sampling is to draw samples from a given probability distribution $p(...
0
votes
0answers
52 views

Gibbs sampler: how can thinning equal to the number of iterations work?

I fit an LDA topic model, using the R package topicmodels. No hiccups and everything runs smoothly, my question here is conceptual. When controlling the Gibbs sampler, the default value (in the ...
1
vote
1answer
459 views

Why Gibbs sampling?

I am new to Gibbs sampling and sampling in general, so here is a basic question. I am reading this tutorial. Equation (40) is our complicated joint probability and equation (49) the less complicated ...
0
votes
1answer
20 views

Direct/indirect sampling of conditionals in Gibbs sampling

I have some problems understanding the definition of Gibbs sampling. Let us take into consideration a bivariate distribution \begin{equation} \pi(x_1,x_2): S \subset \mathcal{R^2} \rightarrow \...
2
votes
1answer
291 views

Transition probabilities for Gibbs Sampling in a Markov Random Field

I am currently reading this paper on Restricted Boltzmann Machines. On page 22, Given a Markov Random Field $\mathbf{X} = (X_1,\ldots,X_N)$ w.r.t a graph $G = (V,E)$ where $V = \{1 \ldots N\}$ and $...
2
votes
0answers
259 views

An efficient way to generate multivariate normal distribution by Gibbs sampler? [closed]

When learning Gibbs sampler, the most used example is bivariate normal. But what if we want to simulate multivariate normal distribution? The computation (mean and variance) of conditional ...
3
votes
0answers
40 views

antagonistic simulated annealing

Simulated annealing aims at a series of target distributions $$\pi_T(x)\propto\exp\{T\,H(x)\}$$ to find the maximum of the function $H$ and its argument $$\arg_x\max_{x\in \mathfrak X} H(x)$$ if the ...
2
votes
0answers
50 views

How to implement a M-H step in a Gibbs sampling

I am having trouble implementing a Metropolis Hastings step in a Gibbs sampling problem. The following code was taken from https://www.stat.colostate.edu/computationalstatistics/ Details: It is a ...
8
votes
1answer
156 views

Would an “importance Gibbs” sampling method work?

I suspect this is a fairly unusual and exploratory question, so please bear with me. I am wondering if one could apply the idea of importance sampling to Gibbs sampling. Here's what I mean: in Gibbs ...
6
votes
3answers
248 views

Conditional distribution of $\exp(-|x|-|y|-a \cdot |x-y|)$

I am trying to use Gibbs sampling or Metropolis-Hastings to draw samples from the joint distribution$$f(x,y)\propto\exp(-|x|-|y|-a \cdot |x-y|)$$ For this I need the conditional distributions of $x$ ...
20
votes
2answers
8k views

Gibbs sampling versus general MH-MCMC

I have just been doing some reading on Gibbs sampling and Metropolis Hastings algorithm and have a couple of questions. As I understand it, in the case of Gibbs sampling, if we have a large ...
7
votes
3answers
2k views

Why does sampling from the posterior predictive distribution $p(x_{new} \mid x_1, \ldots x_n)$ work without having to average out the integral?

In a Bayesian model, the posterior predictive distribution is usually written as: $$ p(x_{new} \mid x_1, \ldots x_n) = \int_{-\infty}^{\infty} p(x_{new}\mid \mu) \ p(\mu \mid x_1, \ldots x_n)d\mu $$ ...
2
votes
1answer
89 views

Gibbs sampler for a multilevel model with no predictors in R

I'm working on multilevel models and want to know how they are estimated in R. For that purpose I'm reading amongst other things "Data Analysis Using Regression and Multilevel/Hierarchical Models" by ...