The Gibbs sampler is a simple form of Markov Chain Monte Carlo simulation, widely used in Bayesian statistics, based on sampling from full conditional distributions for each variable.

learn more… | top users | synonyms

0
votes
0answers
33 views

Gibbs sampling how to sample from the conditional probability? Bayesian model

I want to learn Gibbs sampling for a Bayesian model. How can I sample the variable from the conditional distribution? In this example, arrow means dependent; for example, ...
0
votes
0answers
43 views

Proper likelihood function in acceptance probability of Gibbs Sampler

I have a question about the acceptance ratio used when implementing a random walk M-H in a gibbs sampler to generate sample paths of an unobservable process. When computing the likelihood of a set of ...
1
vote
1answer
50 views

Gibbs sampling with Dirichlet Likelihood

I have a sequence of observations that I am representing as proportions: X1 X2 X3 X4 X5 0.10 0.20 0.50 0.12 0.08 0.07 0.24 0.55 0.04 0.10 ... ...
0
votes
0answers
41 views

how to predict Yn value in this formula with Metropolis Hastings or Gibbs?

I have a model with this formula: $$ Y_n=aX_n^b + e_n $$ $$ X_n \in [0,2] \quad\quad a = 1.5 \quad\quad b = 0.5 \quad \quad e_n = N(μ = 0, σ^2 = 1) $$ I want to predict "$Y_n$" value with using ...
1
vote
0answers
34 views

EM on product of multinomials

I have the following conditional density: $$ P(x | \theta, \pi) = \prod_{i=1}^I \prod_{j=1}^J t_{ij}! \prod_{k=1}^K \frac{1}{x_{ijk}!}(\sum_{l=1}^L \theta_{il} \pi_{jkl})^{x_{ijk}} $$ Here, $x$ is ...
1
vote
1answer
48 views

Gibbs measure and normal distribution

On Wikipedia, the Gibbs measure defines the probability as: $$ P(X=x) = \frac{1}{Z(\beta)}\exp(-\beta E(x)) $$ Now, the familiar form of the normal distribution is: $$ P(x) = ...
0
votes
0answers
43 views

Conditionals for Gibbs sampling of relational clusters

I'm trying to implement a Gibbs sampler, but I'm having trouble to find some of the conditionals of this model. Model We have $A$ actors, $K$ classes or clusters, and a matrix $\phi$ that determines ...
1
vote
1answer
37 views

Gibbs sampler for conditionals that are exponential: Example from Casella & George paper

I am trying to work out Example 2 from Casella and George's paper "Explaining the Gibbs Sampler" in R. The example is: ...
0
votes
0answers
31 views

Joint and Conditional probability

I'm trying to prove a result for Gibbs Sampling with multiple latent variables. I am not sure what the expansion of both the joint and conditional probability would be. In particular, let's say that ...
2
votes
0answers
137 views

Gibbs sampling for LDA — does a small Dirichlet concentration parameter make a difference?

I'm using a Gibbs sampler for Latent Dirichlet allocation as described by Griffiths and Steyvers (http://www.ncbi.nlm.nih.gov/pmc/articles/PMC387300/). The sampling of a new topic $j$ for word $i$ is ...
1
vote
1answer
101 views

How can I estimate the precision of a normal using a Gibbs sampler?

I am trying to estimate the precision $\tau$ of a normal distribution with either WinBUGS or OpenBUGS: $c \sim \text{normal}(\mu,\tau)$ $\mu \rightarrow \lambda \cdot t^{-\beta}$ $\tau \sim ...
2
votes
1answer
76 views

What are the main differences between classical and Gibbs sampling Latent Dirichlet Allocations?

In these weeks I have been studying the classical Latent Dirichlet Allocation (LDA) algorithm by David Blei and colleagues (2003), and the LDA variant based on Gibbs sampling introduced by Tom ...
0
votes
0answers
116 views

Gibbs sampling from the full conditional distribution using R

I have a likelihood function of all the data $y$ $$L(\tau ,\theta |y)\propto \theta ^{\sum \delta _{i}^{C}+\sum \delta _{i}^{H}}\tau ^{\sum \delta _{i}^{H}}e^{-\theta \sum x_{i}^{C}-\tau\theta\sum ...
0
votes
1answer
78 views

Sampling from truncated distribution

I want to sample from a truncated distribution that appears in a Gibbs sampling scheme. The full conditional of the distribution is given by $p(X = k | \ldots) \propto (1 - p)^{k - 1} \mathbb{1} ( s ...
2
votes
0answers
27 views

Gibbs sampler for local linear trend model

Question: Consider the local linear trend model given by: \begin{align*} y_t = \mu_t + \tau \varepsilon_t \ \cdots \ \text{Observation equation} \\ \mu_{t+1} = \phi \mu_t + \eta_t \ \cdots \ ...
3
votes
0answers
41 views

Convergence theorem for Gibbs sampling

The convergence theorem for Gibbs sampling states: Given a random Vektor $X$ with $X_1,X_2,...X_K$ and the knowlegde about the conditional distribution of $X_k$ we can find the actual distribution ...
0
votes
1answer
87 views

Unsupervised Bayesian naive Bayes

I'm reading a paper Gibbs sampling for the uninitiated. In this paper, the authors try to use Gibbs sampling for a bayesian naive bayes model. They formalize the model as a graphical model in page 8. ...
0
votes
1answer
77 views

Implementing Latent Dirichlet Allocation - notation confusion

I am trying to implement LDA using the collapsed Gibbs sampler from http://www.uoguelph.ca/~wdarling/research/papers/TM.pdf the main algorithm is shown below I'm a bit confused about the notation ...
0
votes
0answers
125 views

exponential prior for a gamma distribution

I am planing to make a bayesian inference for a scale mixture of normal distributions. We know the fact the the t-distribution with df $\nu$ can be expressed as a scale mixture of normal ...
0
votes
0answers
56 views

Modeling complex bayes network/MCMC

I would like to try out an idea which is outlined here (http://www.husng.com/content/interpreting-small-sample-sizes-bayesian-estimators), but in a pretty simple model using heuristics. Problem: Lets ...
2
votes
0answers
288 views

Metropolis-Hastings within Gibbs sampling

Suppose we have the following classical normal linear regression model: $$y_i = \beta_1 x_{1i} + \beta_2x_{2i} + \beta_3x_{3i} + e_i$$ where $e_{i} \sim iid.N(0, \sigma^2)$ for all $i = 1, 2, ...
3
votes
1answer
277 views

Gibbs sampling to produce posterior pdf

Suppose we have the following classical normal linear regression model: $$y_i = \beta_1 x_{1i} + \beta_2x_{2i} + \beta_3x_{3i} + e_i$$ where $e_{i} \sim iid.N(0, \sigma^2)$ for all $i = 1, 2, ...
1
vote
1answer
97 views

Sampling from a Deep Belief Network: Treatment of biases in directed part of the model

When generating samples from a DBN, how do you handle the biases that have been learned for the layers below? I know that you normally perform a number of Block Gibbs sampling steps in the undirected ...
5
votes
1answer
151 views

How do programs like BUGS/JAGS automatically determine conditional distributions for Gibbs sampling?

Seems like full conditionals are often quite difficult to derive, yet programs like JAGS and BUGS derive them automatically. Can someone explain how they algorithmically generate full conditionals for ...
1
vote
0answers
66 views

Gibbs sampling in document clustering

I'm trying to understand the idea behind the Topic Models in document clustering. In Latent Dirichlet Allocation, it is necessary to approximate the posterior distribution of topics over the document. ...
-1
votes
2answers
131 views

Any good book for learning probability programming

Are there any good books for me to learn probability programming? For example, I am new to Latent Dirichlet allocation (LDA) and Gibbs sampling. I have read some books about the techniques, but it ...
0
votes
1answer
69 views

How can I sample from the conditional distribution?

I am learning Gibbs Sampling, in which there is a step named sampling from conditional distributions. I don't understand: 1. where is the conditional distribution from? From a general case, how can I ...
0
votes
1answer
106 views

Gibbs Sampling with a single Dirichlet-Multinomial

Simple question: What is the update equation of a $x_i$ in a Gibbs sampling update, $p(x_i | x_{-i})$, if I have the Model: $\theta$ $|$ $\alpha \sim Dir(\alpha)$ $X_i$ $|$ $\theta \sim ...
0
votes
0answers
53 views

How to sample via blocked Gibbs the conjugate normal model?

i am trying to sample the two dimensional parameter $\theta=(\mu,\sigma^2)$, for the Normal model. I have the full conditionals being for the mean: $\pi(\mu_{j}|\ldots) \sim ...
3
votes
1answer
51 views

Confusion related to derivation of conditional distribution

I was reading this paper http://stat.duke.edu/phd-program/gcc/resources/ResourcesDocuments/CodeExplanation.pdf related to Gibbs sampling. Suppose we have n iid samples $x_i$ from $N(\mu,\sigma^2)$ ...
4
votes
1answer
132 views

Confusion related to Gibbs sampling

I came across this article where it says that in Gibbs sampling every sample is accepted. I am a bit confused. How come if every sample it accepted it converges to a stationary distribution. In ...
3
votes
1answer
141 views

Prior selection for Gaussian Processes (GP)

I am trying to select a prior for the covariance parameters of my Gaussian Process (GP) and have been running into numerical problems with my MCMC code. My model is the following: $$Y = D\beta + ...
0
votes
1answer
45 views

Looking for step by step example of sampling from DAG in Bayesian model

I am looking for a tutorial type example that shows the step by step process sampling from a simple hierarchical model. For example, I am trying to study the distribution of ...
2
votes
2answers
776 views

PyMC: how can I define a function of two stochastic variables, with no closed-form distribution?

I'm learning PyMC and basically I have a random variable $Z = X + Y$ where (say) $X \sim \mathrm{Normal}(\theta_X)$ and $Y \sim \mathrm{Lognormal}(\theta_Y)$ and $Z$ has no simple closed-form ...
7
votes
1answer
179 views

Given that one can sample $X \sim f(x)$, is there an easy way to sample $Y \sim k \cdot f(g(y))$ (such as $k \cdot f(e^y)$)?

Say I'm able to sample an RV $X$ from a PDF $f(x)$, can I exploit this to efficiently sample another RV $Y \sim k \cdot f(g(y))$ (where $k$ is a normalizing constant)? I'm interested in something ...
0
votes
0answers
64 views

Gibbs sampling product of normals as conditional

I am deriving a gibbs sampler for a joint distribution, where the conditionals of various parameters are product of two non-standard normal distributions. Usually, I have seen that in Gibbs sampling ...
2
votes
1answer
183 views

Sampling from conditional distribution in general case

I'm dealing with Gibbs Sampling now. Let's consider the example: I know the distribution of X|Y and the distribution of Y. They are some known - Binomial or Beta or other but particular. Thus I have ...
2
votes
1answer
111 views

Calculating conditional probability

I've got the next question. Let's consider I have the pair of distributions: X|t ~ Binomial(n,t); t~Beta(a,b). Here n,a,b are known. I need to construct conditional probability to sample from it, t|X. ...
1
vote
1answer
125 views

Derivation of the posterior over topics in LDA

When studying the latent Dirichlet allocation, I am not very clear about some procedures in their deriving equations. Please refer to the attached figure, how to understand those two steps, marked as ...
2
votes
0answers
134 views

Gibbs sampling from full conditionals

I have the following joint density: $p(x_1,x_2,y_1,y_2) \propto \exp\left(−\left(x_1^2+x_2^2+c_1(y_2-y_1)^2+c_2(y_2-y_1)^4\right)\right)$ Can I use Gibbs sampling to sample from that? How can I get ...
4
votes
1answer
484 views

Bayesian estimation of Dirichlet distribution parameters

I want to estimate parameters of Dirichlet mixture models using Gibbs sampling and I have some questions about that: Is a mixture of Dirichlet distributions equivalent to a Dirichlet process? What ...
1
vote
0answers
79 views

Sampling Stationary Vector Autoregression coefficients while Gibbs Sampling

I have been estimating a Bayesian Vector Autoregression using Gibbs Sampling. When constructing the posterior predictive distribution, I have noticed that when the simulated coefficients from the MCMC ...
1
vote
1answer
124 views

Computing conditional expectation of ordered normal random variables

There are $m$ normally distributed, independent random variables $N_1, \ldots, N_m$ with distinct means $\mu_1, \ldots \mu_m$ and standard deviations $\sigma_1, \ldots, \sigma_m$. Then, we observe a ...
9
votes
1answer
359 views

What are some well known improvements over textbook MCMC algorithms that people use for bayesian inference?

When I'm coding a Monte Carlo simulation for some problem, and the model is simple enough, I use a very basic textbook Gibbs sampling. When it's not possible to use Gibbs sampling, I code the textbook ...
4
votes
2answers
942 views

Generating samples from gibbs sampling

I am quite new to sampling. I am doing Gibbs sampling for a Bayesian network. I am aware about the algorithm for the Gibbs sampling but one thing I am not able to understand. For example let's ...
2
votes
0answers
273 views

Gibbs sampler on the precision (with a gamma prior) in a hierarchical Bayesian model doesn't converge

I am deriving a Gibbs sampler with a model similar to the model in this paper (a graphical model is shown in page 4). To put it simple, my question only concerns $w_i$ (a $K$-dimensional vector drawn ...
1
vote
1answer
268 views

Preparing Bayesian conditional distributions for Gibbs sampling

I was looking at the Gibbs Sampler when I stumbled upon the following example: Suppose $y = (y_{1}, y_{2}, \ldots, y_{n})$ are iid observations from an $N(\mu, \tau^{-1})$ Furthermore, suppose there ...
3
votes
1answer
137 views

What is the state-of-the-art regarding sampling from discrete distribution?

After struggling with auto-Poisson model (a.k.a. Random Markov Network with conditional Poisson distributions) trying to force Gibbs sampler to obtain discrete sample of the network (since I know ...
0
votes
0answers
141 views

Test for convergence within Gibbs sampler

I am running a Gibbs sampler for Multivariate Normal times Inverse Wishart posterior distribution with missing data imputation step. I am trying to check if my step of simulating covariance matrices ...
1
vote
0answers
208 views

Bayesian estimation using Gibbs sampling for financial models

I am trying to do Gibbs sampling, from this paper, www.jds-online.com/file_download/353/JDS-746.pdf This is a CIR financial model, I want to do Gibbs on its parameters: ...