Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [gibbs]

The Gibbs sampler is a simple form of Markov Chain Monte Carlo simulation, widely used in Bayesian statistics, based on sampling from full conditional distributions for each variable or group of variables. The name comes from the method being first used on Gibbs random fields modeling of images by Geman and Geman (1984).

8
votes
0answers
57 views

What is the correct way to write the elastic net?

I am confused about the correct way to write the elastic net. After reading some research papers there seems to be three forms 1) $\exp\{-\lambda_1|\beta_k|-\lambda_2\beta_k^2\}$ 2) $\exp\{-\frac{(\...
0
votes
0answers
23 views

Gibbs sampling for drawing samples and estimating parameters

I'm learning Bayesian inference by myself and having a difficulty for understanding Gibbs sampling. From what I understood, Gibbs sampling is to draw samples from a given probability distribution $p(...
0
votes
0answers
32 views

Gibbs sampler: how can thinning equal to the number of iterations work?

I fit an LDA topic model, using the R package topicmodels. No hiccups and everything runs smoothly, my question here is conceptual. When controlling the Gibbs sampler, the default value (in the ...
0
votes
1answer
14 views

Direct/indirect sampling of conditionals in Gibbs sampling

I have some problems understanding the definition of Gibbs sampling. Let us take into consideration a bivariate distribution \begin{equation} \pi(x_1,x_2): S \subset \mathcal{R^2} \rightarrow \...
2
votes
0answers
114 views

An efficient way to generate multivariate normal distribution by Gibbs sampler? [closed]

When learning Gibbs sampler, the most used example is bivariate normal. But what if we want to simulate multivariate normal distribution? The computation (mean and variance) of conditional ...
3
votes
0answers
29 views

antagonistic simulated annealing

Simulated annealing aims at a series of target distributions $$\pi_T(x)\propto\exp\{T\,H(x)\}$$ to find the maximum of the function $H$ and its argument $$\arg_x\max_{x\in \mathfrak X} H(x)$$ if the ...
2
votes
0answers
41 views

How to implement a M-H step in a Gibbs sampling

I am having trouble implementing a Metropolis Hastings step in a Gibbs sampling problem. The following code was taken from https://www.stat.colostate.edu/computationalstatistics/ Details: It is a ...
2
votes
1answer
72 views

Gibbs sampler for a multilevel model with no predictors in R

I'm working on multilevel models and want to know how they are estimated in R. For that purpose I'm reading amongst other things "Data Analysis Using Regression and Multilevel/Hierarchical Models" by ...
0
votes
0answers
23 views

Gibbs sampling where I can only find the mode of conditionals?

I'm trying to solve a problem with Gibbs Sampling, so I'm trying to do: $$ x_1^1 \sim p(x_1 | x_2^0, x_3^0)\\ x_2^1 \sim p(x_2 | x_1^1, x_3^0)\\ x_3^1 \sim p(x_3 | x_1^1, x_2^1)\\ x_1^2 \sim p(x_1 | ...
1
vote
0answers
25 views

Finding the posterior distribution of mean and variance given data sample using Gibbs Sampling?

I have the following hierachical bayesian model - $\mathbf{x}|\mathbf{c},\sigma^2 \sim \mathcal{N}(\mathbf{x}|\mathbf{c},\sigma^2)$ $\mathbf{c}|\mathbf{c}_1,\sigma^2_2 \sim \mathcal{N}(\mathbf{c}|\...
8
votes
1answer
144 views

Would an “importance Gibbs” sampling method work?

I suspect this is a fairly unusual and exploratory question, so please bear with me. I am wondering if one could apply the idea of importance sampling to Gibbs sampling. Here's what I mean: in Gibbs ...
0
votes
0answers
21 views

Determine frequent states

I have a distribution from which I can sample (namely, a Boltzmann Machine). Which methods exists to determine frequent states (states with high probability) / the most frequent state (state with the ...
0
votes
0answers
23 views

forward sampling for Bayesian network with continuous variables and equation-based causal relationships

I have a physical system which can be represented by the following Bayesian network. It has the following characteristics 1) The encoded variables are continuous variables 2) The causal ...
0
votes
0answers
82 views

Gibbs sampler for ARIMA AR(1) parameters: division by zero

Suppose the following AR(1) model: $$ y_t = \mu + \phi (y_{t-1} - \mu) + \epsilon_t $$ with $\epsilon_t \sim \mathcal{N}(0,\sigma^2)$. Following issue arises when sampling from $P(\mu_i \;|\; \phi_{...
5
votes
3answers
232 views

Conditional distribution of $\exp(-|x|-|y|-a \cdot |x-y|)$

I am trying to use Gibbs sampling or Metropolis-Hastings to draw samples from the joint distribution$$f(x,y)\propto\exp(-|x|-|y|-a \cdot |x-y|)$$ For this I need the conditional distributions of $x$ ...
2
votes
1answer
39 views

Gibbs sampling allocations for time dependent observations from this model

I observe $N$ observations $\{x_{1,t_1}, \dots, x_{N,t_N}\}$ from a $k$ component Gaussian Mixture model. The $i$th observation is seen at time stamp $t_i$ and is distributed such that each $x_{i,t_i}|...
1
vote
0answers
23 views

Gibbs Sampler for mixture models: shall I skip some samples to avoid to use correlated samples? [duplicate]

I am implementing a Gibbs sampler in order to estimate the parameters of a mixture model. Assuming that the parameters are contained in a vector $\theta$ what I will do is: Implement and run the ...
0
votes
0answers
40 views

bayesian decision making - comparing expected loss

The problem is like this: Suppose that I am considering which country should I invest on, country A and country B, based on their GDP growth rate $\alpha$. There are two possible choices for each ...
0
votes
1answer
128 views

Inferring GMM parameters with Gibbs Sampling

On my book, "Machine Learning A Probabilistic Approach". It's stated that is straightforward to derive a Gibbs sampling algorithm to fit a mixture model, especially if we use conjugate priors. So ...
1
vote
1answer
102 views

Conditional distribution in this Gaussian Mixture Model

Say I observe $N$ observations $\{x_1, \dots, x_N\}$ from a $k$ component Gaussian Mixture model, with $k > 0$ known and is such that each $x_i|\boldsymbol{\pi}, \boldsymbol{\mu} \sim \sum_{j=1}^{k}...
1
vote
1answer
60 views

Gibbs sampler for Dirichlet Process concentration parameter

I am trying to implement a Gibbs sampler for Hierarchical Dirichlet process, but I cannot seem to correctly estimate the concentration parameters. I therefore started testing just this part of a ...
2
votes
1answer
84 views

Conditional Distribution to sample

Suppose I have six data points (n,x): (14,5), (13,4), (7,3), (10,5), (12,7), (20,13) which are realizations of binomial experiments on n trials with x successes respectively.. And I assume I ...
11
votes
2answers
615 views

Is Gibbs sampling an MCMC method?

As far as I understand it, it is (at least, that is how Wikipedia defines it). But I've found this statement by Efron* (emphasis added): Markov chain Monte Carlo (MCMC) is the great success story ...
0
votes
1answer
146 views

How to create a distribution and sample?

Suppose we are given some small set of data on bundles of electrical wires and increasing voltages run through them, and we note how many of the individual wires fail. So for example, a large data ...
1
vote
0answers
30 views

How exactly does Gibbs sampling work in Markov Networks?

I was going through the Probabilistic Graphical Modelling course by Stanford and they used a network such as this one-https://imgur.com/gallery/k0C8FY2 Now if we want to sample P(A|B), how would we ...
2
votes
1answer
269 views

Conditional distribution for Gibbs sampling for Gaussian mixture

If we draw $n$ i.i.d. points $x_1,x_2,\dots,x_n$ from the following Gaussian mixture: $$ \frac 12 \mathcal N(x \mid \mu_1,1) + \frac 12 \mathcal N(x\mid \mu_2,1) $$ and the prior $p(\mu_1 , \mu_2 )$ ...
1
vote
1answer
43 views

Sampling from partial posterior

I'm reading a paper where the authors have something like the following as one step in their MCMC: $$ \begin{align} y&=\rho_1 x_1+\rho_2 x_2+\epsilon\\ z&=\beta\tilde{y}+u \end{align} $$ ...
1
vote
0answers
39 views

HDP: Gibbs sampler implementation

I am trying to recreate the model proposed by Gao et al. (2011), based on the Hierarchical Dirichlet Process proposed by Teh and al. (2005). To estimate the model (let's call it iHDP) I need to ...
2
votes
0answers
89 views

Can Bayesian Optimization solve this problem?

Suppose ${\bf{x}} = (x_1,\ldots,x_n)$ and $f({\bf{x}})\propto 1_A({\bf{x}}) \prod_{i=1}^n {x_i}^{\alpha_i-1} e^{-\beta_i x_i}$ , i.e. $f$ is proportional to the product of independent gamma ...
4
votes
1answer
37 views

Find joint maximum of a sampled density

I used a sampling method to fit a model with three parameters to data, by supplying the likelihood function and priors. (I'm using JAGS but I think this applies to any method). I obtain triplets of ...
0
votes
1answer
365 views

What is the difference between monte carlo integration and gibbs sampling?

I am aware that both are methods of sampling from the posterior. MC integration replaces the integral by a sample MC sample. Is this sample independent? Gibbs sampling is a class of MCMC ...
2
votes
1answer
28 views

Slice Sampling asks to draw from $f^{-1}]y,+\infty[$

Slice Sampling asks to draw uniformly from $f^{-1}]y,+\infty[$. Wikipedia page However, how can we be sure that a uniform defined over the set $f^{-1}]y,+\infty[$ is in fact proper? If I had to ...
3
votes
1answer
143 views

Importance weight of conditioned particle in conditional SMC

In a generic particle filter, I understand the importance weights for each particle are calculated as $w_t^s \propto w_{t-1}^s \frac{p(y_t \mid z_t^s) p(z_t^s \mid z_{t-1}^s)}{q(z_t^s \mid z_{t-1}^s, ...
15
votes
1answer
1k views

Stan $\hat{R}$ versus Gelman-Rubin $\hat{R}$ definition

I was going through the Stan documentation which can be downloaded from here. I was particularly interested in their implementation of the Gelman-Rubin diagnostic. The original paper Gelman & ...
3
votes
1answer
72 views

Sampling posterior of empty cluster in GMM and Gibbs

Consider performing inference via a standard Gibbs sampler for a standard Gaussian Mixture Model (GMM) with $k$ components that are Gaussians $$\mathcal{N}(\mu_{k}, \sigma^{2}_{k})$$ where we assume ...
0
votes
1answer
40 views

Distribution sampling when no analytic expression

The goal of gibbs sampling is to sample the joint distribution when this latter has not an analytic expression, by deriving the conditional distribution of each variable. So it is supposed that the ...
4
votes
2answers
372 views

Posterior computation for Laplace distribution

I am dealing with being Bayesian and looking for a closed form for a posterior for the scale parameter $\tau$ of a Laplace distribution, such that I can derive a full conditional in my Gibbs sampler. ...
2
votes
2answers
68 views

Generating samples from Gibbs method

I have a following homework in a subject called "Monte-Carlo Methods". I would be very thankful, if you could help me with this one, because I'm a bit stuck with this one .. The task is as follows:...
3
votes
1answer
78 views

Gibbs Sampling vs. Using Raw Probability in Contrastive Divergence

In Hinton's Practical Guide to Training Restricted Boltzmann Machines, Section 3, he discusses different situations in which one should take a sample from the Gibbs sampling process, and other ...
2
votes
0answers
124 views

Bayesian prior and posterior computation for a truncated normal

I have to deal with data in a Bayesian framework, ultimately devising a Gibbs sampler for inferring all my distributions parameters. Specifically, suppose I observe some univariate data distributed ...
1
vote
0answers
50 views

Non-Identifiable Multivariate Normal Posterior

So I have a theoretical question about what looks like, in my opinion, a multivariate normal distribution. The issue comes with the fact that the data is distributed with likelihood: Y |θ1, θ2 ∼ N(θ1 ...
2
votes
1answer
157 views

Why can Gibbs sampling outputs be used in Rao-Blackwellization?

I'm currently learning Chib (1995)'s method to calculate the marginal likelihood of a Bayesian model using Gibbs sampling outputs. I'm stuck in the Rao-Blackwellization step. Suppose $\mu$ and $\phi$...
3
votes
0answers
181 views

Is burn-in necessary for MCMC/Gibbs sampling if I have samples from the true distribution already?

Say I have some samples from a distribution $p$, and I want to get more samples using MCMC/Gibbs sampling. Since the existing samples are known from the equilibrium distribution $p$, if I use them as ...
2
votes
1answer
71 views

Why does $P(\theta_1\mid D, \theta_2) \propto P(D \mid \theta_1, \theta_2)P(\theta_1)$ hold?

Suppose that in a Bayesian framework we have observed data $D$, using independent prior distributions on the parameters of the model, denoted by $\theta_1, \theta_2$. Then, the joint posterior ...
0
votes
1answer
101 views

prior for initial values of Kalman Filter

I'm studying Carter and Kohn's (1994) implementation of the Gibbs sampler for Bayesian analysis of state space models. In their paper, they assume the starting value, call it $\beta_0$, of the state ...
5
votes
1answer
79 views

How to use Gibbs sampling when target function is known only up to normalising constant?

Assume we want to use Gibbs sampling to get an estimate of the parameter , and that we have the following expression for the conditional posterior of : If I am not mistaken, this means that the ...
1
vote
0answers
54 views

help interpreting plot of MCMC sample

I am estimating a model using MCMC (Gibbs Sampling). Because of the complexity of the model, I have been running two chains with many iterations. A plot of the draws for each parameter reveals a ...
3
votes
2answers
271 views

Gibbs sampling and Bayesian inference

I wanted to get a more in-depth understanding of sampling algorithms, and so I thought I could start with the very simple example of a binomial or bernoulli likelihood with a beta prior (since it is ...
1
vote
0answers
69 views

Burning and Sampling in Gibbs sampler [closed]

I am confused about the concept of sampling in the Gibbs sampler after the burn-in loops. This is the basic problem, I have a picture composed of 1's and 0's. This is a noisy version and I am trying ...
1
vote
1answer
299 views

When can the collapsed Gibbs sampler be implemented?

I understand Gibbs sampling is a means of statistics inference, and it seems that sometimes certain variables can be integrated out in the sampling process, known as collapsed Gibbs sampling. I really ...