1
$\begingroup$

Consider two asymmetric Laplace distribution $L_1(\mu_1,\sigma_1,\tau_1)$ and $L_2(\mu_2,\sigma_2,\tau_2)$ where

\begin{equation} L(x;\mu,\sigma,\tau) =\frac{\tau(1-\tau)}{\sigma} \begin{cases} \text{exp}(- \frac{x-\mu}{\sigma}(\tau - 1)) & \text{if $x < \mu$}\\ \text{exp}(-\frac{x-\mu}{\sigma}\tau) & \text{if $x \ge \mu$} \end{cases}\\ =\frac{\tau(1-\tau)}{\sigma}\text{exp}(-\rho_{\tau}(\frac{x-\mu}{\sigma})) \end{equation}

where $\rho_{\tau}(\frac{x-\mu}{\sigma}) = \frac{x-\mu}{\sigma}(\tau - \mathbb{1}(x<\mu) )$

I'm wondering if there is an closed form solution to the KL-divergence between two laplaces ?

\begin{equation} \mathbb{E}_{L_1}[\text{log}(L_1) - \text{log}(L_2)]=\text{log}(\frac{\tau_1}{\tau_2}) + \text{log}(\frac{1-\tau_1}{1-\tau_2}) + \text{log}(\frac{\sigma_1}{\sigma_2}) + \mathbb{E}_{L_1}[T] \end{equation}

where

\begin{equation} T = \rho_{\tau_2}(\frac{x-\mu_2}{\sigma_2}) -\rho_{\tau_1}(\frac{x-\mu_1}{\sigma_1})\\ = \frac{x-\mu_2}{\sigma_2}(\tau_2 - \mathbb{1}(x<\mu_2)) - \frac{x-\mu_1}{\sigma_1}(\tau_1 - \mathbb{1}(x<\mu_1))\\ =\frac{\tau_2}{\sigma_2}x - \frac{\tau_2\mu_2}{\sigma_2} - \frac{\mathbb{1}(x<\mu_2)}{\sigma_2}x + \mathbb{1}(x<\mu_2)\frac{\mu_2}{\sigma_2} - \frac{\tau_1}{\sigma_1}x + \frac{\tau_1\mu_1}{\sigma_1} + \frac{\mathbb{1}(x<\mu_1)}{\sigma_1}x -\mathbb{1}(x<\mu_1)\frac{\mu_1}{\sigma_1} \end{equation}

So now; by linearity of expectation, $\mathbb{E}_{L_1}[T]$ leads to compute

$$\frac{\tau_2}{\sigma_2}\mathbb{E}_{L_1}[x] - \frac{\tau_2\mu_2}{\sigma_2} - \frac{1}{\sigma_2}\mathbb{E}_{L_1}[x\mathbb{1}(x<\mu_2)] + \mathbb{E}_{L_1}[\mathbb{1}(x<\mu_2)]\frac{\mu_2}{\sigma_2} - \frac{\tau_1}{\sigma_1}\mathbb{E}_{L_1}[x] + \frac{\tau_1\mu_1}{\sigma_1} + \frac{1}{\sigma_1}\mathbb{E}_{L_1}[x\mathbb{1}(x<\mu_1)] -\mathbb{E}_{L_1}[\mathbb{1}(x<\mu_1)]\frac{\mu_1}{\sigma_1}$$

where $ \mathbb{E}_{L_1}[\mathbb{1}(x<\mu_1)] = \mathbb{P}(x<\mu_1),\mathbb{E}_{L_1}[\mathbb{1}(x<\mu_2)] = \mathbb{P}(x<\mu_2)$; the CDF of $L_1$ Laplace distribution given by https://en.wikipedia.org/wiki/Asymmetric_Laplace_distribution when $\lambda = \frac{\sqrt{\tau(1-\tau)}}{\sigma}$ and $\kappa = \sqrt{\frac{\tau}{1-\tau}}$

Now; it remains to compute $\mathbb{E}_{L_1}[x\mathbb{1}(x<\mu_1)]$ and $\mathbb{E}_{L_1}[x\mathbb{1}(x<\mu_2)]$

$\endgroup$
  • 3
    $\begingroup$ Have you set the integral expressing the K-L divergence between two of these distributions? When did it get intractable? Have you checked using a formal integrator like Wolfram's? $\endgroup$ – Xi'an Nov 18 '19 at 17:11
  • $\begingroup$ @Xi'an; Hi thanks for the reply; so I follow your step and wondering for the last step; is it correct so far? $\endgroup$ – ElleryL Nov 18 '19 at 22:17
  • $\begingroup$ @Xi'an; Hi thanks for the hint; so I re-write the distribution in easier way (to get rid of the integration on different cases); but I'm not sure if I'm correct. I did write the KL in term of the mean if $L_1$ as you hinted. Do you mind give a check ? Thanks a lot ! $\endgroup$ – ElleryL Nov 19 '19 at 19:47
  • $\begingroup$ @Xi'an; you're right; since we integrating over all $x$ and indicator doesn't swap outside. But I couldn't simplify the correct one in the form of $\mu_1, \mu_2$ especially the density function itself also contains indicator function $\endgroup$ – ElleryL Nov 19 '19 at 22:40
  • $\begingroup$ @Xi'an; Hi I've found one error in $T$ where there should be $\mathbb{E}[\mathbb{1}(x < \cdot)]$; I'm not sure if this is what you're referring to ? It requires additional computation of CDF of laplace distribution; which I re-edited in the post. $\endgroup$ – ElleryL Nov 20 '19 at 15:12
2
$\begingroup$

Expectations are derived by standard exponential integrations \begin{align*}\mathbb E_{L_1}[\mathbb 1(X<\mu_1)]&=\int_{-\infty}^{\mu_1} \frac{\tau_1(1-\tau_1)}{\sigma_1}\text{exp}\left(- \frac{x-\mu_1}{\sigma_1}(\tau_1 - 1)\right)\text{d}x\\ &=\tau_1\int_{-\infty}^{\mu_1}\frac{(1-\tau_1)}{\sigma_1}\text{exp}\left(\frac{1-\tau_1}{\sigma_1}(x-\mu_1)\right)\text{d}x \\ &=\tau_1\int_0^\infty \frac{(1-\tau_1)}{\sigma_1}\text{exp}\left(-\frac{1-\tau_1}{\sigma_1}y\right)\text{d}y \\ &=\tau_1 \end{align*} \begin{align*}\mathbb E_{L_1}[X\mathbb 1(X<\mu_1)]&=\int_{-\infty}^{\mu_1}x \frac{\tau_1(1-\tau_1)}{\sigma_1}\text{exp}\left(- \frac{x-\mu_1}{\sigma_1}(\tau_1 - 1)\right)\text{d}x\\ &=\tau_1\int_{-\infty}^{\mu_1}x\frac{(1-\tau_1)}{\sigma_1}\text{exp}\left(\frac{x-\mu_1}{\sigma_1}(1-\tau_1)\right)\text{d}x \\ &=\tau_1\int_{-\infty}^{\mu_1}(x-\mu_1+\mu_1)\frac{(1-\tau_1)}{\sigma_1}\text{exp}\left(\frac{x-\mu_1}{\sigma_1}(1-\tau_1)\right)\text{d}x \\ &=\mu_1\mathbb E_{L_1}[\mathbb 1(X<\mu_1)]+\tau_1\int_0^{\infty}y\frac{(1-\tau_1)}{\sigma_1}\text{exp}\left(-\frac{1-\tau_1}{\sigma_1}y\right)\text{d}\\ &=\mu_1\tau_1+\tau_1\frac{\sigma_1}{1-\tau_1} \end{align*} \begin{align*}\mathbb E_{L_1}[\mathbb 1(X<\mu_2)]&=\mathbb E_{L_1}[\mathbb 1(X<\mu_1)]+\mathbb E_{L_1}[\mathbb 1(\mu_1<X<\mu_2)]\qquad\qquad(\mu_1<\mu_2)\\ &=\tau_1+\int_{\mu_1}^{\mu_2} \frac{\tau_1(1-\tau_1)}{\sigma_1}\text{exp}\left(- \frac{\tau_1}{\sigma_1} (x-\mu_1)\right)\text{d}x\\ &=\tau_1+(1-\tau_1)\int_0^{\mu_2-\mu_1} \frac{\tau_1}{\sigma_1}\text{exp}\left(-\frac{\tau_1}{\sigma_1}y\right)\text{d}y \\ &=\tau_1+(1-\tau_1)\left\{1-\exp(-\tau_1(\mu_2-\mu_1)/\sigma_1) \right\} \end{align*} \begin{align*}\mathbb E_{L_1}[X\mathbb 1(X<\mu_2)]&=\mathbb E_{L_1}[X\mathbb 1(X<\mu_1)]+\mathbb E_{L_1}[X\mathbb 1(\mu_1<X<\mu_2)]\qquad\qquad(\mu_1<\mu_2)\\ &=\tau_1\left\{\mu_1+\frac{\sigma_1}{1-\tau_1}\right\}+\int_{\mu_1}^{\mu_2} x\frac{\tau_1(1-\tau_1)}{\sigma_1}\text{exp}\left(- \frac{\tau_1}{\sigma_1} (x-\mu_1)\right)\text{d}x\\ &=\tau_1\left\{\mu_1+\frac{\sigma_1}{1-\tau_1}\right\}+(1-\tau_1)\int_0^{\mu_2-\mu_1} (y+\mu_1)\frac{\tau_1}{\sigma_1}\text{exp}\left(-\frac{\tau_1}{\sigma_1}y\right)\text{d}y \\ &=\tau_1\left\{\mu_1+\frac{\sigma_1}{1-\tau_1}\right\}+(1-\tau_1)\mu_1\left\{1-\exp(-\tau_1(\mu_2-\mu_1)/\sigma_1) \right\}\\ &\quad+(1-\tau_1)\big\{-(\mu_2-\mu_1)\exp(-\tau_1(\mu_2-\mu_1)/\sigma_1)\\ &\qquad\left.+\frac{\sigma_1}{\tau_1}[1-\exp(-\tau_1(\mu_2-\mu_1)/\sigma_1)]\right\} \end{align*} Alternatively, most of these computations stem from the (basic) observation that an asymmetric Laplace distribution $\mathcal L^a(\mu,\tau,\sigma)$ is a mixture of two drifted Exponential distributions$$\tau \left\{\mu-\mathcal E([1-\tau]/\sigma)\right\}+(1-\tau)\left\{\mu+\mathcal E(\tau/\sigma)\right\}$$

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.