3
$\begingroup$

I'd like to prove what concentration parameter $\kappa \in [0, \infty)$ maximizes the (differential) entropy of a von-Mises Fisher Distribution.

The differential entropy of of a von Mises-Fisher distribution $vMF(\mu, \kappa)$ is:

$$-\log C_D(\kappa) - \kappa A_D(\kappa)$$

where

  • $C_D(\kappa) = \frac{\kappa^{D/2 - 1}}{(2 \pi)^{D/2} I_{D/2 - 1}(\kappa)}$
  • $A_D(\kappa) = \frac{I_{D/2}(\kappa)}{I_{D/2 - 1}(\kappa)}$
  • $I_v$ denotes the modified Bessel function of the first kind at order $v$

My intuition is that $\kappa=0$ maximizes the entropy. But how do I show this? I'm having trouble calculating first and second order stationary points because of the Bessel functions.

$\endgroup$
5
  • 2
    $\begingroup$ Shouldn't it be the lowest possible $\kappa$? $\endgroup$
    – Neil G
    Commented Aug 25, 2023 at 6:34
  • $\begingroup$ That's my guess too, yes. I wrote my intuition is that $\kappa = 0$ maximizes the entropy but I'm having trouble deriving this. Do you know how to show this? I rephrased to sharpen what specifically I'm asking. $\endgroup$ Commented Aug 25, 2023 at 6:42
  • $\begingroup$ You could plug the expression into an automatic differentiator and see if you can show that the differential entropy is monotonic wrt $\kappa$. $\endgroup$
    – Neil G
    Commented Aug 25, 2023 at 7:08
  • 1
    $\begingroup$ This is a special case of a general result: the entropy among all distributions on a space of finite volume is maximized by the uniform distribution. Thus, it shouldn't be necessary to use any of the particulars about Bessel functions. Use the Calculus of Variations to show the log density must be almost surely zero. $\endgroup$
    – whuber
    Commented Aug 25, 2023 at 16:58
  • $\begingroup$ @whuber yes, $p$ should have been $D$. My apologies. There's no need to use "??" $\endgroup$ Commented Aug 25, 2023 at 19:08

1 Answer 1

4
$\begingroup$

This post presents an elementary but highly general demonstration that avoids any considerations of Bessel functions. (You need to know some basic derivatives of real functions of one variable, as presented in the first section, and only the simplest properties of integrals -- linearity and positivity--; no integrals will need to be evaluated.)

Calculus preliminaries

Observe that the function defined on $(0,\infty)$ (and extended to $0$ via continuity) as

$$u(x) = x\log x$$

is strictly convex because $u^{\prime\prime}(x) = 1/x$ is everywhere positive. Because whenever $x\gt 0$ also $1+x\gt 0,$ Jensen's Inequality implies

$$\frac{1+x}{2} \log\left(\frac{1+x}{2}\right) = u\left(\frac{1+x}{2}\right) \lt \frac{1}{2}u(1) + \frac{1}{2}u(x) = \frac{1}{2}u(x).$$

Next, $u(x) \ge x-1$ for all $x \ge 0$ because

$$\frac{\mathrm d}{\mathrm d x}(u(x) - (x-1)) = \log x$$

has a unique zero at $x=1,$ which must be the point of the global minimum, and

$$u(1) - (1-1) = 1\log(1) - 0 = 0$$

is its minimal value, QED.

A general result

Let $\Omega$ be a measure space and $\mu$ be a finite measure on $\Omega.$ Divide $\mu$ by $\mu(\Omega)$ to normalize it.

Suppose that $f$ is a density function with respect to $\mu$ for which $\int_\Omega f\,\mathrm d\mu = 1:$ that is, $f$ is a probability density. We are interested in the functional

$$H[f] = \int_\Omega f \log f \,\mathrm d\mu,$$

the negative of the entropy. The preceding inequality shows that $H$ is nonnegative since

$$H[f] = H[f] + 1 - 1 = H[f] + \int_\Omega (f\log f - f + 1)\,\mathrm d\mu\ge \int_\Omega 0 \,\mathrm d\mu = 0.$$

Moreover, the constant function $1$ attains this minimum because

$$H[1] = \int_\Omega 1 \log 1\,\mathrm d\mu = \int_\Omega 0 \,\mathrm d\mu = 0.$$

What was the effect of the initial normalization of $\mu$? It did two things:

  1. All integrals were multiplied by $1/\mu(\Omega).$

  2. All density functions had to be multiplied by $\mu(\Omega)$ to compensate for (1).

The first effect will not change the fact that $0$ is the lower bound of $H,$ while the second effect changes $H$ to

$$\begin{aligned} H[\mu(\Omega)f] &= \int_\Omega \mu(\Omega) f \log(\mu(\Omega) f)\,\mathrm d\mu\\ &= \mu(\Omega) \int_\Omega f \log f\,\mathrm d\mu + \mu(\Omega)\log(\mu(\Omega))\int_\Omega f\,\mathrm d\mu\\ &= \mu(\Omega) H[f] + \mu(\Omega)\log(\mu(\Omega)) \end{aligned}$$

because $\int_\Omega f\,\mathrm d\mu = 1.$ This constant additive change will not alter the order relationship between $H[1]$ and $H[f]$ -- it only changes the value of the lower bound.

(It is instructive to compare this to the analysis at https://stats.stackexchange.com/a/415436/919 which concerns probability distributions on a non-finite measure space. There, the analog of changing the measure is to rescale the variable.)

We conclude

On a space $\Omega$ of finite measure, the entropy $-H[f]$ is maximized among probability densities $f$ for the constant probability density $f = 1/\mu(\Omega)$ (attaining the minimum value $\mu(\Omega)\log\mu(\Omega)$).

Solution to the problem

A von-Mises Fisher distribution in $D$ (Euclidean) dimensions is an isotropic Normal distribution conditioned on the unit sphere. (The parameter $1/\kappa$ uniformly scales that distribution.) Applying the preceding result to the induced Lebesgue measure $\mu$ on the sphere shows that the entropy is maximized by the uniform distribution on the sphere. That is the distribution corresponding to $\kappa = 0,$ QED.

$\endgroup$
8
  • 1
    $\begingroup$ This is amazing - thank you!! Indeed a general result, and one that also nicely answers my question :) $\endgroup$ Commented Aug 25, 2023 at 19:14
  • 1
    $\begingroup$ Sensible (although I didn't expect it until you mentioned it) that there is the possibility of a non-zero minimum entropy. (+1) $\endgroup$
    – Galen
    Commented Aug 25, 2023 at 19:30
  • 1
    $\begingroup$ @Galen That gave me some trouble ;-). I initially solved this by observing that the map $f\to(1+f)/2$ on the normalized space is a contraction on the complete metric space $\mathcal L^1(\Omega,\mu),$ which therefore has a unique stationary point, but I also needed this map to increase the entropy. What I find interesting is that by normalizing the space we guarantee the differential entropy always is nonnegative and is zero only for the uniform distribution. In this fashion the differential entropy on such spaces becomes almost as nice as the entropy on discrete spaces. $\endgroup$
    – whuber
    Commented Aug 25, 2023 at 19:37
  • 1
    $\begingroup$ @Galen Yes. My resource for that is Rosenlicht's Introduction to Analysis. (It, along with Spivak's Calculus on Manifolds, was the textbook for my first college math course.) $\endgroup$
    – whuber
    Commented Aug 25, 2023 at 20:03
  • 1
    $\begingroup$ @Galen I can't find it on my shelves right now, but likely the first part of Chapter VIII on fixed point theorems is what I was recalling. $\endgroup$
    – whuber
    Commented Aug 25, 2023 at 20:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.