# KL divergence between two Asymmetric Laplace distributions?

Consider two asymmetric Laplace distribution $$L_1(\mu_1,\sigma_1,\tau_1)$$ and $$L_2(\mu_2,\sigma_2,\tau_2)$$ where

$$$$L(x;\mu,\sigma,\tau) =\frac{\tau(1-\tau)}{\sigma} \begin{cases} \text{exp}(- \frac{x-\mu}{\sigma}(\tau - 1)) & \text{if x < \mu}\\ \text{exp}(-\frac{x-\mu}{\sigma}\tau) & \text{if x \ge \mu} \end{cases}\\ =\frac{\tau(1-\tau)}{\sigma}\text{exp}(-\rho_{\tau}(\frac{x-\mu}{\sigma}))$$$$

where $$\rho_{\tau}(\frac{x-\mu}{\sigma}) = \frac{x-\mu}{\sigma}(\tau - \mathbb{1}(x<\mu) )$$

I'm wondering if there is an closed form solution to the KL-divergence between two laplaces ?

$$$$\mathbb{E}_{L_1}[\text{log}(L_1) - \text{log}(L_2)]=\text{log}(\frac{\tau_1}{\tau_2}) + \text{log}(\frac{1-\tau_1}{1-\tau_2}) + \text{log}(\frac{\sigma_1}{\sigma_2}) + \mathbb{E}_{L_1}[T]$$$$

where

$$$$T = \rho_{\tau_2}(\frac{x-\mu_2}{\sigma_2}) -\rho_{\tau_1}(\frac{x-\mu_1}{\sigma_1})\\ = \frac{x-\mu_2}{\sigma_2}(\tau_2 - \mathbb{1}(x<\mu_2)) - \frac{x-\mu_1}{\sigma_1}(\tau_1 - \mathbb{1}(x<\mu_1))\\ =\frac{\tau_2}{\sigma_2}x - \frac{\tau_2\mu_2}{\sigma_2} - \frac{\mathbb{1}(x<\mu_2)}{\sigma_2}x + \mathbb{1}(x<\mu_2)\frac{\mu_2}{\sigma_2} - \frac{\tau_1}{\sigma_1}x + \frac{\tau_1\mu_1}{\sigma_1} + \frac{\mathbb{1}(x<\mu_1)}{\sigma_1}x -\mathbb{1}(x<\mu_1)\frac{\mu_1}{\sigma_1}$$$$

So now; by linearity of expectation, $$\mathbb{E}_{L_1}[T]$$ leads to compute

$$\frac{\tau_2}{\sigma_2}\mathbb{E}_{L_1}[x] - \frac{\tau_2\mu_2}{\sigma_2} - \frac{1}{\sigma_2}\mathbb{E}_{L_1}[x\mathbb{1}(x<\mu_2)] + \mathbb{E}_{L_1}[\mathbb{1}(x<\mu_2)]\frac{\mu_2}{\sigma_2} - \frac{\tau_1}{\sigma_1}\mathbb{E}_{L_1}[x] + \frac{\tau_1\mu_1}{\sigma_1} + \frac{1}{\sigma_1}\mathbb{E}_{L_1}[x\mathbb{1}(x<\mu_1)] -\mathbb{E}_{L_1}[\mathbb{1}(x<\mu_1)]\frac{\mu_1}{\sigma_1}$$

where $$\mathbb{E}_{L_1}[\mathbb{1}(x<\mu_1)] = \mathbb{P}(x<\mu_1),\mathbb{E}_{L_1}[\mathbb{1}(x<\mu_2)] = \mathbb{P}(x<\mu_2)$$; the CDF of $$L_1$$ Laplace distribution given by https://en.wikipedia.org/wiki/Asymmetric_Laplace_distribution when $$\lambda = \frac{\sqrt{\tau(1-\tau)}}{\sigma}$$ and $$\kappa = \sqrt{\frac{\tau}{1-\tau}}$$

Now; it remains to compute $$\mathbb{E}_{L_1}[x\mathbb{1}(x<\mu_1)]$$ and $$\mathbb{E}_{L_1}[x\mathbb{1}(x<\mu_2)]$$

• Have you set the integral expressing the K-L divergence between two of these distributions? When did it get intractable? Have you checked using a formal integrator like Wolfram's? – Xi'an Nov 18 '19 at 17:11
• @Xi'an; Hi thanks for the reply; so I follow your step and wondering for the last step; is it correct so far? – ElleryL Nov 18 '19 at 22:17
• @Xi'an; Hi thanks for the hint; so I re-write the distribution in easier way (to get rid of the integration on different cases); but I'm not sure if I'm correct. I did write the KL in term of the mean if $L_1$ as you hinted. Do you mind give a check ? Thanks a lot ! – ElleryL Nov 19 '19 at 19:47
• @Xi'an; you're right; since we integrating over all $x$ and indicator doesn't swap outside. But I couldn't simplify the correct one in the form of $\mu_1, \mu_2$ especially the density function itself also contains indicator function – ElleryL Nov 19 '19 at 22:40
• @Xi'an; Hi I've found one error in $T$ where there should be $\mathbb{E}[\mathbb{1}(x < \cdot)]$; I'm not sure if this is what you're referring to ? It requires additional computation of CDF of laplace distribution; which I re-edited in the post. – ElleryL Nov 20 '19 at 15:12

Expectations are derived by standard exponential integrations \begin{align*}\mathbb E_{L_1}[\mathbb 1(X<\mu_1)]&=\int_{-\infty}^{\mu_1} \frac{\tau_1(1-\tau_1)}{\sigma_1}\text{exp}\left(- \frac{x-\mu_1}{\sigma_1}(\tau_1 - 1)\right)\text{d}x\\ &=\tau_1\int_{-\infty}^{\mu_1}\frac{(1-\tau_1)}{\sigma_1}\text{exp}\left(\frac{1-\tau_1}{\sigma_1}(x-\mu_1)\right)\text{d}x \\ &=\tau_1\int_0^\infty \frac{(1-\tau_1)}{\sigma_1}\text{exp}\left(-\frac{1-\tau_1}{\sigma_1}y\right)\text{d}y \\ &=\tau_1 \end{align*} \begin{align*}\mathbb E_{L_1}[X\mathbb 1(X<\mu_1)]&=\int_{-\infty}^{\mu_1}x \frac{\tau_1(1-\tau_1)}{\sigma_1}\text{exp}\left(- \frac{x-\mu_1}{\sigma_1}(\tau_1 - 1)\right)\text{d}x\\ &=\tau_1\int_{-\infty}^{\mu_1}x\frac{(1-\tau_1)}{\sigma_1}\text{exp}\left(\frac{x-\mu_1}{\sigma_1}(1-\tau_1)\right)\text{d}x \\ &=\tau_1\int_{-\infty}^{\mu_1}(x-\mu_1+\mu_1)\frac{(1-\tau_1)}{\sigma_1}\text{exp}\left(\frac{x-\mu_1}{\sigma_1}(1-\tau_1)\right)\text{d}x \\ &=\mu_1\mathbb E_{L_1}[\mathbb 1(X<\mu_1)]+\tau_1\int_0^{\infty}y\frac{(1-\tau_1)}{\sigma_1}\text{exp}\left(-\frac{1-\tau_1}{\sigma_1}y\right)\text{d}\\ &=\mu_1\tau_1+\tau_1\frac{\sigma_1}{1-\tau_1} \end{align*} \begin{align*}\mathbb E_{L_1}[\mathbb 1(X<\mu_2)]&=\mathbb E_{L_1}[\mathbb 1(X<\mu_1)]+\mathbb E_{L_1}[\mathbb 1(\mu_1 \begin{align*}\mathbb E_{L_1}[X\mathbb 1(X<\mu_2)]&=\mathbb E_{L_1}[X\mathbb 1(X<\mu_1)]+\mathbb E_{L_1}[X\mathbb 1(\mu_1 Alternatively, most of these computations stem from the (basic) observation that an asymmetric Laplace distribution $$\mathcal L^a(\mu,\tau,\sigma)$$ is a mixture of two drifted Exponential distributions$$\tau \left\{\mu-\mathcal E([1-\tau]/\sigma)\right\}+(1-\tau)\left\{\mu+\mathcal E(\tau/\sigma)\right\}$$