I am dealing with being Bayesian and looking for a closed form for a posterior for the scale parameter $\tau$ of a Laplace distribution, such that I can derive a full conditional in my Gibbs sampler.

I do not think I could exploit conjugacy for the Laplace, but any closed-form computation for the posterior would be useful. Any ideas on how to choose the prior and obtain a closed form posterior from which it is known how to sample from?

  • 9
    $\begingroup$ To "deal with being a Bayesian," have you considered joining a Bayesian support group? :-) $\endgroup$ – whuber Apr 30 '18 at 14:16
  • 2
    $\begingroup$ The scale parameter should be conjugate with the inverse gamma distribution. $\endgroup$ – aleshing Apr 30 '18 at 15:46
  • $\begingroup$ @marmle do you have a reference and a derivation for that? $\endgroup$ – rano Apr 30 '18 at 15:50

The likelihood for $n$ iid observations looks like:

$ f(x_1,...x_n|\lambda,\mu) \propto \frac{1}{\lambda^n} exp(-\frac{1}{\lambda}\sum_{i=1}^n|x_i-\mu|)$

Hence a conjugate prior for $\lambda$ with $\mu, x$ known must (thinking only about the algebra) look like:

$ f(\lambda) \propto \frac{1}{\lambda^a} exp(-\frac{b}{\lambda})$

As suggested by marmle, this is an Inverse Gamma, although to be nice we'd need to change $a\rightarrow a-1$ and let $a>0, b>0$.

EDIT: To get the updated parameters for $\lambda$:

$ f(\lambda|x_1,...x_n, \mu) \propto f(\lambda)f(x_1,...x_n|\lambda,\mu) $

$ \propto \frac{1}{\lambda^{a-1}} exp(-\frac{b}{\lambda}) \frac{1}{\lambda^n} exp(-\frac{1}{\lambda}\sum_{i=1}^n|x_i-\mu|)$

$ \propto \frac{1}{\lambda^{n+a-1}} exp(-\frac{1}{\lambda}(b+\sum_{i=1}^n|x_i-\mu|)) $

  • $\begingroup$ do you have a reference for this and how to update the parameters for the posterior? $\endgroup$ – rano Apr 30 '18 at 16:23
  • $\begingroup$ No but that's the derivation, so I guess it depends on what you need it for. $\endgroup$ – conjectures Apr 30 '18 at 17:05
  • $\begingroup$ I need it to sample from efficiently. By looking at the classic pdf of an inverse Gamma, as described in here: en.wikipedia.org/wiki/Inverse-gamma_distribution , do you consider the $\frac{b^{a}}{\Gamma(a)}$ to be the normalizing factor? $\endgroup$ – rano Apr 30 '18 at 17:29
  • $\begingroup$ Would then be possibile to put a Normal prior on $\mu$ and obtain a NormalInverseGamma conjugate prior, as it is for the classical Normal parameters $\mu$ and $\sigma$? $\endgroup$ – rano Apr 30 '18 at 17:45
  • 2
    $\begingroup$ Sorry, to be clear, you read off the updated params as the expression that is in the same place as the prior, e.g $a$ goes to $n+a$. The location parameter isn't conjugate using the normal distribution. In fact, it is not obvious what it should be (due to absolute value). $\endgroup$ – conjectures Apr 30 '18 at 18:34

Conditioning on $\lambda$, the posterior on $\mu$ can be expressed in a reasonably closed form: \begin{align} \pi(\mu|x_1,\ldots,x_n) &\propto \pi(\mu) \exp\left(-\frac{1}{\lambda}\sum_{i=1}^n|x_i-\mu|\right)\\ &= \pi(\mu) \exp\left(-\frac{1}{\lambda}\sum_{i=1}^n|x_{(i)}-\mu|\right)\\ &= \pi(\mu) \sum_{j=0}^n \mathbb{I}_{(x_{(j)},x_{(j+1)})}(\mu) \exp\left(-\frac{1}{\lambda}\sum_{i=1}^n|x_{(i)}-\mu|\right)\\ &= \pi(\mu) \sum_{j=0}^n \mathbb{I}_{(x_{(j)},x_{(j+1)})}(\mu) \exp\left(\frac{1}{\lambda}\sum_{i=1}^j[x_{(i)}-\mu|]\right)\exp\left(\frac{1}{\lambda}\sum_{i=j+1}^n[\mu-x_{(i)}|]\right)\\ &=\sum_{j=0}^n \exp\left(\frac{1}{\lambda}\sum_{i=1}^jx_{(i)}-\frac{1}{\lambda}\sum_{i=j+1}^nx_{(i)}\right)\pi(\mu)\exp\left(\frac{2j-n}{\lambda}\mu\right)\mathbb{I}_{(x_{(j)},x_{(j+1)})}(\mu) \end{align} since using a Normal prior $\pi(\mu)$ returns a mixture of truncated Normals. For instance, if the prior is a ${\cal N}(0,\sigma^2)$, then \begin{align} \pi(\mu|x_1,\ldots,x_n) &\propto \sum_{j=0}^n \overbrace{\exp\left(\frac{1}{\lambda}\sum_{i=1}^jx_{(i)}-\frac{1}{\lambda}\sum_{i=j+1}^nx_{(i)}\right)}^{\omega_j}\exp\left(\frac{2j-n}{\lambda}\mu-\frac{\mu^2}{2\sigma^2}\right)\mathbb{I}_{(x_{(j)},x_{(j+1)})}(\mu)\\ &\propto \sum_{j=0}^n \omega_j\exp\left(2\sigma^2\frac{2j-n}{\lambda}\frac{\mu}{2\sigma^2}-\frac{\mu^2}{2\sigma^2}\right)\mathbb{I}_{(x_{(j)},x_{(j+1)})}(\mu)\\ &\propto \sum_{j=0}^n \omega_j\exp\left(\sigma^2\frac{(2j-n)^2}{\lambda^2}\right)\exp\left(\frac{-1}{2\sigma^2}\left[\mu-\frac{(2n-j)\sigma^2}{\lambda}\right]^2\right) \mathbb{I}_{(x_{(j)},x_{(j+1)})}(\mu)\\ \end{align} a mixture of $n+1$ truncated Normal distributions in $\mu$, truncated respectively to the intervals $(x_{(j)},x_{(j+1)})$ with original mean $(2n-j)\sigma^2/\lambda$ and original variance $\sigma^2$. (While obvious, the weights of the mixture are cumbersome in that they imply the coverage probability of $(x_{(j)},x_{(j+1)})$ by the Normal distribution with mean $(2n-j)\sigma^2/\lambda$ and variance $\sigma^2$.)

And the conditional posterior on $\lambda$ associated with an Inverse Gamma ${\cal IG}(a,b)$ is indeed an Inverse Gamma$${\cal IG}\left(a+n,b+\sum_i|x_i-\mu|\right)$$which implies that a two-step Gibbs sampler can be implemented.

  • $\begingroup$ could you please elicit where the mixture of truncated normals lies and how to update its parameters? $\endgroup$ – rano May 2 '18 at 20:23
  • $\begingroup$ @rano: sorry, I missed this comment of yours. In the final line of the expression of $\pi(\mu|x_1,\ldots)$ I obtained a sum of $n+1$ terms, each involving a Normal density, a weight (independent from $\mu$), and an indicator function on $\mu$. The product of the density with the indicator produces a truncated Normal density, with the weight being corrected accordingly. Since this is the posterior distribution, there is no reason to "update its parameters", unless you are contemplating a sequential update of the posterior, feasible albeit dreary: modify all weights and replace one component. $\endgroup$ – Xi'an Aug 4 '20 at 8:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.