Questions tagged [gibbs]

The Gibbs sampler is a simple form of Markov Chain Monte Carlo simulation, widely used in Bayesian statistics, based on sampling from full conditional distributions for each variable or group of variables. The name comes from the method being first used on Gibbs random fields modeling of images by Geman and Geman (1984).

Filter by
Sorted by
Tagged with
2
votes
1answer
247 views

What does it means of Normalization term of Gibbs distribution?

I am studying about Gibbs distribution concept and I am confusing a one term in that concept that is normalization term. According to the Hammersley–Clifford theorem, an random $x$ can equivalently be ...
1
vote
0answers
11 views

Gibbs sampling Bayesian conditional distribution for mean of a Normal distribution

first post here in CV. I'm currently working on a textbook exercise on Gibbs Sampling and got stuck on naming the distribution for one of the conditional distributions. Question Consider a normal ...
2
votes
0answers
34 views

Gibbs sampler of a generative model

I understand what a Gibbs sampler is and I understand how LDA does classification. But I'm unsure how I can generate a Gibbs sampler for an LDA model and how to meld the two concepts. Let's say I ...
3
votes
1answer
47 views

Bayesian mixture model joint posterior

I am just starting to learn about bayesian mixture models. There is a few clarifications that I want to make which I am not sure myself. The graphical model below describes a gaussian mixture model ...
3
votes
0answers
47 views

Implementation of a blocked Gibbs sampler for a mixture model with a Dirichlet-process prior

I am trying to understand and implement the blocked Gibbs sampler described on page 552 in Bayesian Data Analysis by Gelman et al. in the context of using a Dirichlet process as a prior in a mixture ...
1
vote
2answers
60 views

Deriving full conditionals from joint distributions?

In this link (https://www.youtube.com/watch?v=a_08GKWHFWo), the author derives the conditional distributions from the joint; but I got lost in the mechanics of what happened, the process was overly ...
1
vote
0answers
19 views

Gibbs sampling proposals for bivariate normal?

I'm very familiar with Metropolis-Hastings, having implemented the algorithm myself to handle "toy problems." Gibbs sampling, however, is a bit trickier for me as I'm not quite certain what ...
4
votes
1answer
121 views

Sampling from multivariate normal conditional on a negative minimum

Let $X\sim \mathcal{N}(\mu,\Sigma)$, where $\mu\in\mathbb{R}^n$ and $\Sigma\in\mathbb{R}^{n\times n}$. How can I efficiently sample from $X | {\min{X}\le 0}$? (I.e. from the distribution of $X$ ...
1
vote
0answers
21 views

Intuition on why Gibbs Sampling samples from the posterior distribution

I am new to Gibbs Sampling and I do understand how the algorithm works but I would also like to understand how sampling from the conditional distributions is equivalent to sampling from the joint. ...
0
votes
0answers
11 views

Uncorrelated Samples from a non-conjugate (but well behaved) posterior

I'm trying to create a Dirichlet process mixture model with a kernel distribution similar to a product of gammas. (in fact, if I generate a latent random variable, it IS a product of (independent) ...
0
votes
1answer
165 views

How to create a distribution and sample?

Suppose we are given some small set of data on bundles of electrical wires and increasing voltages run through them, and we note how many of the individual wires fail. So for example, a large data ...
3
votes
1answer
124 views

Gibbs Sampling vs. Using Raw Probability in Contrastive Divergence

In Hinton's Practical Guide to Training Restricted Boltzmann Machines, Section 3, he discusses different situations in which one should take a sample from the Gibbs sampling process, and other ...
0
votes
0answers
14 views

How does pymc3 posterior simulation work in this simple case without having the full conditional distributions?

I'm trying to estimate the posterior distribution of the gamma parameters alpha and beta given that my data comes from a gamma distribution and the priors I chose come from two uniform distributions. ...
0
votes
1answer
28 views

Running several MCMC chains after convergence?

I am running a MCMC Gibbs sampler for a computationally expensive model. It takes ~12 hours to obtain 1000 iterations of this MCMC sampler. I have tested the sampler, and I found that the chain seems ...
0
votes
0answers
33 views

Estimating parameters with Gibbs sampling?

I've been trying to understand Gibbs sampling; my end goal is to intuitively understand it in the context of MCMC methods. However, in order to reach that end, I started a with simpler example. I ...
0
votes
0answers
14 views

Stationary distibution of Gibbs sampler

I am suppose to find stationary distribution of a Marcov chain generated by the following Gibbs sampler $n := 0$; $X_0 = x_0$; $(x_0>0)$ repeat Gen $Y_n \sim U(0, \exp(-X_n))$; Gen $X_{n+1} \...
0
votes
0answers
30 views

python gibbs sampler for bivariate normal distribution, failing to converge

I've been trying to understand Gibbs sampling for some time. Recently, I saw a video that made a good deal of sense. https://www.youtube.com/watch?v=a_08GKWHFWo The author used Gibbs sampling to ...
4
votes
2answers
68 views

What is the relationship between Boltzmann / Gibbs sampling and the softmax function?

I'm looking at sampling functions in the context of reinforcement learning; specifically the explore/exploit problem. A method I've seen pretty often is to derive the action by assigning a score to ...
0
votes
1answer
36 views

Why does my Gibbs sampler find two optimals?

[EDITED] I am using a Gibbs Sampler to find a Bayesian optimization to my multilevel (hierarchical) model (2 levels). However, when I run multiple chains (each chain having different starting values) ...
0
votes
1answer
33 views

Gibbs Sampler Conditional Marginal Computation

I have the following question regarding the Gibbs sampler, although it might be considered a simple question on conditional probability. For sake of simplicity, let us say we are trying to sample ...
1
vote
1answer
36 views

Relationship between variational inference and sampling in a Boltmzann-machine-like network

In this paper concerning a Boltzmann-machine-like network and its variational mean field approximation, the authors write In the stochastic system as well as the deterministic system, units evolve ...
0
votes
0answers
8 views

Inference on Author model

The Author Model is an LDA based model that first time introduced in paper [The Author-Topic Model for Authors and Documents]. I have studied the inference of the LDA model and know how to obtain the ...
1
vote
0answers
24 views

parameter estimation on the LDA model

I have a problem with estimating the parameters of $\theta$, and $\phi$ in the Latent Dirichlet Allocation (LDA) model. The article Finding scientific topics has done the estimation of the parameters ...
-1
votes
1answer
22 views

Which sampling methods (MCMC or otherwise) can be used if the posterior distribution is unknown?

The goal is to sample the posterior distribution of parameters describing some model (fairly low dimensional, generally no more than 10 parameters at the absolute most, usually around 5), but I don't ...
0
votes
0answers
24 views

What pitfalls should we avoid with Heidelberger-Welch convergence

I'm working through validating a Bayesian mixture model for multi-species occupancy with a collaborator. Initially, we relied on coda::heidel.diag to alert us to ...
1
vote
1answer
38 views

Multi-steps direct forecasting in AR(2) model through bayesian estimation of the model

I'm estimating an AR(2) model using Bayesian methods through Gibbs sampling and I want to perform 4 step ahead multi-steps direct forecasts. Inside the MCMC loop in each iteration I'm drawing the ...
0
votes
1answer
62 views

Replicating an experiment on GMRF (Gaussian Markov Random Field)

I am trying to understand an experiment from this paper, specifically Section 5.2. In the paper, they propose a new algorithm for computing the log-determinant of sparse matrices, and in section 5 ...
0
votes
0answers
20 views

How to calculate marginal transition matrix given the joint probability distribution for Gibbs sampling?

If I have two variables x$_{1}$ and x$_{2}$ which both take values in {1,2,3}, and I have the table representing their joint probabilities, how can I then determine the transition matrix for the ...
2
votes
1answer
92 views

What should be the burn in period for Metropolis-within-Gibbs?

I need to get samples from an unnormalized distribution $p(\theta, \tau | D)$. However, sampling directly from the joint distribution with Metropolis-Hastings is hard, as the sampler rarely finds ...
1
vote
0answers
19 views

Gibbs sampler with adaptive linear transformation

It is a well known fact that linear transformations can dramatically improve the performances of a Gibbs sampler when a ridge-like joint likelihood function occurs. Can I make an algorithm that ...
0
votes
0answers
34 views

Building an hierarchical model

I am trying to understand hierarchic Bayesian models. I am following the example http://www.openbugs.net/Examples/Pumps.html. The exercise I am trying to complete relates to the same problem, with the ...
0
votes
0answers
30 views

Sampling from hyperprior with Gibbs sampling

Let's say I have some set of data, $\vec{y}$, where each element is sampled as $$ y_n \sim Normal(\mu,1/\tau) $$ where $\tau$ is the precision, $1/\sigma^2$, and I want to use Gibbs sampling, choosing ...
1
vote
1answer
45 views

Gibbs sampling example of a bivariate normal with unknown correlation

I'm looking for an example of using Gibbs sampling with a bivariate normal, where the correlation parameter is not fixed or known. In other words, what is the conditional distribution of the ...
2
votes
1answer
55 views

Gibbs sampling for Multivariate: how to update?

In this page of Murphy's 'Machine Learning: a Probabilistic Perspective' it's explained how to do Gibbs sampling on a Gaussian Mixture Model. Reading this, I was trying to understand when to update ...
3
votes
2answers
85 views

R alternatives to JAGS/BUGS [closed]

I've recently fit more complex hidden markov models with random effects and covariates etc. JAGS was the only program that could get the job done. Now I want to write my own functions to facilitate ...
1
vote
1answer
66 views

Stationarity of coefficients when sampling VAR by Gibbs sampler

I am using Gibbs Sampler for VAR and have noticed that some researchers check the stationarity of $\beta$ coefficients while drawing. I am not sure why do they do that? Bayesian VARs do not require ...
0
votes
0answers
46 views

Derivation of a Gibbs sampler for a Bayesian model with hierarchical Dirichlet prior

I am studying Gibbs sampling. In particular, I got stuck on deriving Gibbs sampling when the reference Bayesian model has hierarchical Dirichlet distributions. As an example, let us start with the ...
2
votes
1answer
49 views

Gibbs Sampling - Calculating the full conditionals from the joint density

Given a joint density, $f(x_1, x_2)$, can its pmf/pdf be found generally by the method outlined below: For a joint density, $f(x_1, x_2)$ if we hold $x_2$ constant in the joint density, we will get ...
3
votes
1answer
116 views

Do I need to evaluate acceptance rates in Metropolis within Gibbs algorithm?

Consider the Gibbs sampler Sample $\theta' \sim p(\theta|\tau, D)$ Sample $\tau' \sim p(\tau|\theta', D)$ where $\theta,\tau$ parameters of the data $D$. Now assume that we can only sample from $p(\...
1
vote
0answers
32 views

Sampling states of an “unnatural” Hamiltonian System

I would like to sample from a Gibbs distribution given by $$f(p, q) = \frac{1}{\mathcal{Z}}e^{-H(p, q; \omega, J)}$$ where $H$ is the Hamiltonian on generalized coordinates $(p,q)\in \mathbb{R}^{2n}...
2
votes
2answers
203 views

How does this Sampler work for the Concentration parameter of Dirichlet Process?

I am puzzled by how this Gibbs sampler on section 6 of Escobar & West (1995) works. To put it in simple words, the aim is to sample $\alpha$. The defined terms are: $$\eta\sim \texttt{Beta}(a,b)$$ ...
1
vote
0answers
22 views

Why the nodes in a Boltzmann machine need to be sampled one at a time?

Typically, we use Gibbs sampling to update (or generate samples from) energy based models. This means we update each node while keeping its markov blanket constant. Why can't we update/sample all ...
0
votes
1answer
169 views

What does MCMC do during burn-in period?

I am studying mcmc and I am wondering what mcmc does during burn-in period. And also what is the difference during burning period and after the burn-in period?
1
vote
1answer
117 views

What is the role of simulated annealing in Gibbs sampling?

While I was reading about Gibbs sampling, I happened to see "simulated annealing" but what is it doing in Gibbs sampling? Although I don't understand the full context of simulated annealing, I am ...
0
votes
0answers
26 views

Does MCMC Gibbs sampling algorithm first build a steady Markov Chain, then does the sampling to build the posterior distribution?

I am currently studying MCMC Gibbs sampling and while reading this part, a question has come into my head if MCMC Gibbs sampling first build a steady Markov Chain and does the sampling or does ...
2
votes
2answers
69 views

Need to understand a statement for Random Walk Metropolis algorithm's proposal distribution?

I was told that the proposal distribution of Random Walk Metropolis needs to be symmetric. But today I was reading a book about Bayesian Analysis which contains the following statement: "The proposal ...
1
vote
0answers
184 views

Bayesian Gamma Regression Update

I'm looking for a resource that explains how to do update the coefficients for a Bayesian gamma regression using Gibbs sampling. Specifically, if $y_i \sim Gamma(\alpha,\beta_i)$ and my data ...
40
votes
1answer
15k views

What is the difference between Metropolis-Hastings, Gibbs, Importance, and Rejection sampling?

I have been trying to learn MCMC methods and have come across Metropolis-Hastings, Gibbs, Importance, and Rejection sampling. While some of these differences are obvious, i.e., how Gibbs is a special ...
0
votes
0answers
28 views

Gibbs sampling for simple posterior distribution?

I have a likelihood function, $$ p(x) = \theta^{\sum x} (1- \theta)^{n-\sum x} $$ and prior distribution, $$ p(\theta) \propto \theta^{\alpha - 1} (1- \theta)^{\beta - 1}$$ then the posterior ...
-1
votes
1answer
413 views

Inferring GMM parameters with Gibbs Sampling

On my book, "Machine Learning A Probabilistic Approach". It's stated that is straightforward to derive a Gibbs sampling algorithm to fit a mixture model, especially if we use conjugate priors. So ...

1
2 3 4 5
8