I was reading the discussion on Hacker News about the use of the standard deviation as opposed to other metrics such as the mean absolute deviation. So, if we were to follow the principle of maximum entropy, what kind of distribution would we use with if we only knew the distribution's mean and the mean absolute deviation?

Or does it make more sense to use the median and the mean absolute deviation from the median?

I found a paper Maximum Entropy Principle with General Deviation Measures by Grechuk, Molyboha and Zabarankin which appears to have the information I am curious about, but it is taking me a while to decipher it.

  • $\begingroup$ Interesting question; welcome to Cross Validated! $\endgroup$ – Nick Stauner Jan 16 '14 at 0:52

These wise gentlemen, Kotz, S., Kozubowski, T. J., & Podgorski, K. (2001). The Laplace Distribution and Generalizations: A Revisit with Applications to Communications, Economics, Engineering, and Finance (No. 183). Springer.

challenge us with an exercise:

enter image description here

The proof can follow the Information-Theoretic proof that the Normal is maximum entropy for given mean and variance. Specifically: Let $f(x)$ be the above Laplace density, and let $g(x)$ be any other density, but having the same mean and mean absolute deviation. This means that the following equality holds:

$$E_g(|X-c_1|)=\int g(x)|x-c_1|dx=c_2 =\int f(x)|x-c_1|dx =E_f(|X-c_1|)\qquad [1]$$ Now consider the Kullback-Leibler Divergence of the two densities:

$$0\le D_{KL}(g||f) = \int g(x)\ln\left(\frac {g(x)}{f(x)}\right)dx = \int g(x)\ln g(x)dx -\int g(x)\ln f(x)dx \qquad [2]$$

The first integral is the negative of the (differential) entropy of $g$, denote it $-h(g)$. The second integral is (writing explicitly the Laplacian pdf)

$$\int g(x)\ln[f(x)]dx = \int g(x)\ln\left[\frac{1}{2c_2}\exp\left\{-\frac 1{c_2} |x-c_1|\right\}\right]dx $$ $$=\ln\left[\frac{1}{2c_2}\right]\int g(x)dx- \frac 1{c_2}\int g(x)|x-c_1|dx $$ The first integral integrates to unity, and using also eq. $[1]$ we obtain

$$\int g(x)\ln[f(x)]dx = -\ln\left[2c_2\right] - \frac 1{c_2}\int f(x)|x-c_1|dx = -(\ln\left[2c_2\right] +1) $$ But this is the negative of the differential entropy of the Laplacian, denote it $-h(f)$.

Inserting these results into eq. $[2]$ we have $$0\le D(g||f) = -h(g) - (-h(f)) \Rightarrow h(g) \le h(f)$$ Since $g$ was arbitrary, this proves that the above Laplacian density is maximum entropy among all distributions with the above prescriptions.

  • $\begingroup$ Such a simple distribution, and a nice write-up too! I suspected the distribution would be smooth except at 0. $\endgroup$ – Dietrich Epp Feb 5 '14 at 2:53
  • $\begingroup$ Thanks. Sometime "same goes with same" -so since the Laplace distribution involves the absolute value, it was a prime suspect. $\endgroup$ – Alecos Papadopoulos Feb 5 '14 at 10:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.