13
$\begingroup$

Does there exist a conjugate prior for the Laplace distribution? If not, is there a known closed form expression that approximates the posterior for the parameters of the Laplace distribution?

I've googled around quite a lot with no success so my current guess is "no" on the questions above...

$\endgroup$
  • 1
    $\begingroup$ Google "polson and scott normal variance mean mixtures" - this will give you some approximate bayes using MAP via em algorithm. $\endgroup$ – probabilityislogic Feb 4 '14 at 12:11
12
$\begingroup$

Let's look at them one at a time first (taking the other as given).

From the link (with the modification of following the convention of using Greek symbols for parameters):

$f(x|\mu,\tau) = \frac{1}{2\tau} \exp \left( -\frac{|x-\mu|}{\tau} \right) \,$

- scale parameter:

$\cal{L}(\tau) \propto \tau^{-k-1} e^{-\frac{S}{\tau}} \,$

for certain values of $k$ and $S$. That is the likelihood is of inverse-gamma form.

So the scale parameter has a conjugate prior - by inspection the conjugate prior is inverse gamma.

- location parameter

This is, indeed, more tricky, because $\sum_i|x_i-\mu|$ doesn't simplify into something convenient in $\mu$; I don't think there's any way to 'collect the terms' (well in a way there sort of is, but we don't need to anyway).

A uniform prior will simply truncate the posterior, which isn't so bad to work with if that seems plausible as a prior.

One interesting possibility that may occasionally be useful is it's rather easy to include a Laplace prior (one with the same scale as the data) by use of a pseudo-observation. One might also approximate some other (tighter) prior via several pseudo-observations)

In fact, to generalize from that, if I were working with a Laplace, I'd be tempted to simply generalize from constant-scale-constant-weight to working with a weighted-observation version of Laplace (equivalently, a potentially different scale for every data point) - the log-likelihood is still just a continuous piecewise linear function, but the slope can change by non-integer amounts at the join points. Then a convenient "conjugate" prior exists - just another 'weighted' Laplace or, indeed, anything of the form $\exp(-\sum_j |\mu-\theta_j|/\phi_j)$ or $\exp(-\sum_j w^*_j|\mu-\theta_j|)$ (though it would need to be appropriately scaled to make an actual density) - a very flexible family of distributions, and which apparently results in a posterior "of the same form" as the weighted-observation likelihood, and something easy to work with and draw; indeed even the pseudo-observation thing still works.

It is also flexible enough that it can be used to approximate other priors.

(More generally still, one could work on the log-scale and use a continuous, piece-wise-linear log-concave prior and the posterior would also be of that form; this would include asymmetric Laplace as a special case)

Example

Just to show that it's pretty easy to deal with - below is a prior (dotted grey), likelihood (dashed, black) and posterior (solid, red) for the location parameter for a weighted Laplace (... this was with known scales).

enter image description here

The weighted Laplace approach would work nicely in MCMC, I think.

--

I wonder if the resulting posterior's mode is a weighted median?

-- actually (to answer my own question), it looks like the answer to that is 'yes'. That makes it rather nice to work with.

--

Joint prior

The obvious approach would be to write $f(\mu,\tau)=f(\mu|\tau)f(\tau)$: it would be relatively easy to have $\mu|\tau$ in the same form as above - where $\tau$ could be a scaling factor on the prior, so the prior would be specified relative to $\tau$ - and then an inverse gamma prior on $\tau$, unconditionally.

Doubtless something more general for the joint prior is quite possible, but I don't think I'll pursue the joint case further than that here.

--

I've never seen or heard of this weighted-laplace prior approach before, but it was rather simple to come up with so it's probably been done already. (References are welcome, if anyone knows of any.)

If nobody knows of any references at all, maybe I should write something up, but that would be astonishing.

$\endgroup$
  • $\begingroup$ Wow, great answer. I sure don't know any references to anything similar. If you find something, or write something up, please let me know! $\endgroup$ – Rasmus Bååth Feb 4 '14 at 9:20
  • 1
    $\begingroup$ One possible way to get at the location parameter is to use the normal variance mixture representation of laplace. This is a conditionally conjugate prior though... $\endgroup$ – probabilityislogic Feb 4 '14 at 12:07
  • $\begingroup$ @probabilityislogic that's interesting. In earlier edits, I put in a line pointing out that the Laplace was an exponential scale mixture of normals because I wondered if there might be something could be done with that, but as I edited the answer further it no longer fit anywhere and I took it out again. From your helpful comment it sounds like it can be used in that way; that's likely to be handy. $\endgroup$ – Glen_b -Reinstate Monica Feb 4 '14 at 16:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.